{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0YEWQgiBU7gh"
   },
   "outputs": [],
   "source": [
    "# Copyright 2021 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_title:migration,automl,icn"
   },
   "source": [
    "# AutoML SDK: AutoML tables regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JE-aKjayJjkF"
   },
   "source": [
    "## Installation\n",
    "\n",
    "Install the latest version of AutoML SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7dxB6-B0JjkG"
   },
   "outputs": [],
   "source": [
    "! pip3 install google-cloud-automl\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Iddo-8HmJjkH"
   },
   "source": [
    "Install the Google *cloud-storage* library as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xHvi6EFSJjkH"
   },
   "outputs": [],
   "source": [
    "! pip3 install google-cloud-storage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ktRTgB8DJjkI"
   },
   "source": [
    "### Restart the Kernel\n",
    "\n",
    "Once you've installed the AutoML SDK and Google *cloud-storage*, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "x1g4mBdlJjkI"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "\n",
    "if not os.getenv(\"AUTORUN\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pxAMbzEgfD7e"
   },
   "source": [
    "## Before you begin\r\n",
    "\r\n",
    "### GPU run-time\r\n",
    "\r\n",
    "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\r\n",
    "\r\n",
    "### Set up your GCP project\r\n",
    "\r\n",
    "**The following steps are required, regardless of your notebook environment.**\r\n",
    "\r\n",
    "1. [Select or create a GCP project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\r\n",
    "\r\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\r\n",
    "\r\n",
    "3. [Enable the AutoML APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\r\n",
    "\r\n",
    "4. [Google Cloud SDK](https://cloud.google.com/sdk) is already installed in AutoML Notebooks.\r\n",
    "\r\n",
    "5. Enter your project ID in the cell below. Then run the  cell to make sure the\r\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\r\n",
    "\r\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hceMthWGJjkI"
   },
   "source": [
    "#### Project ID\n",
    "\n",
    "**If you don't know your project ID**, try to get your project ID using `gcloud` command by executing the second cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "set_project_id"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"[your-project-id]\" #@param {type:\"string\"}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "autoset_project_id",
    "outputId": "2663b1ca-f6c6-43ed-c23b-654919cb200c"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
    "    # Get your GCP project id from gcloud\n",
    "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID:\", PROJECT_ID)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0UfsdLCEJjkJ"
   },
   "outputs": [],
   "source": [
    "! gcloud config set project $PROJECT_ID\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jT0fijejJjkK"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook.  Below are regions supported for AutoML. We recommend when possible, to choose the region closest to you.\n",
    "\n",
    "- Americas: `us-central1`\n",
    "- Europe: `europe-west4`\n",
    "- Asia Pacific: `asia-east1`\n",
    "\n",
    "You cannot use a Multi-Regional Storage bucket for training with AutoML. Not all regions provide support for all AutoML services. For the latest support per region, see [Region support for AutoML services]()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CTBlncfrJjkK"
   },
   "outputs": [],
   "source": [
    "REGION = 'us-central1' #@param {type: \"string\"}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J_croPBoJjkK"
   },
   "source": [
    "#### Timestamp\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append onto the name of resources which will be created in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KKxStk5bJjkL"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R0InH8s1f-uL"
   },
   "source": [
    "### Authenticate your GCP account\r\n",
    "\r\n",
    "**If you are using AutoML Notebooks**, your environment is already\r\n",
    "authenticated. Skip this step.\r\n",
    "\r\n",
    "*Note: If you are on an AutoML notebook and run the cell, the cell knows to skip executing the authentication steps.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6aUuFIftxXiN"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "# If you are running this notebook in Colab, run this cell and follow the\n",
    "# instructions to authenticate your Google Cloud account. This provides access\n",
    "# to your Cloud Storage bucket and lets you submit training jobs and prediction\n",
    "# requests.\n",
    "\n",
    "# If on AutoML, then don't execute this code\n",
    "if not os.path.exists('/opt/deeplearning/metadata/env_version'):\n",
    "    if 'google.colab' in sys.modules:\n",
    "        from google.colab import auth as google_auth\n",
    "        google_auth.authenticate_user()\n",
    "\n",
    "    # If you are running this tutorial in a notebook locally, replace the string\n",
    "    # below with the path to your service account key and run this cell to\n",
    "    # authenticate your Google Cloud account.\n",
    "    else:\n",
    "        %env GOOGLE_APPLICATION_CREDENTIALS your_path_to_credentials.json\n",
    "\n",
    "    # Log in to your account on Google Cloud\n",
    "    ! gcloud auth login\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MDUAZaN3JjkL"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "This tutorial is designed to use training data that is in a public Cloud Storage bucket and a local Cloud Storage bucket for your batch predictions. You may alternatively use your own training data that you have stored in a local Cloud Storage bucket.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bucket"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"[your-bucket-name]\" #@param {type:\"string\"}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "autoset_bucket"
   },
   "outputs": [],
   "source": [
    "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
    "    BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yU6VuylGJjkM"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QVRixM4oJjkM"
   },
   "outputs": [],
   "source": [
    "! gsutil mb -l $REGION gs://$BUCKET_NAME\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mi-Pdh6QJjkN"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aqDptKpYJjkN"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al gs://$BUCKET_NAME\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fR9geV9pJjkO"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "Next, set up some variables used throughout the tutorial.\n",
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hIHTX-pkJjkO"
   },
   "source": [
    "#### Import AutoML SDK\n",
    "\n",
    "Import the AutoM SDK into our Python environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zbd7zoxhU7he"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "from google.cloud import automl_v1beta1 as automl\n",
    "from google.protobuf.json_format import MessageToJson, ParseDict\n",
    "from google.protobuf.struct_pb2 import Value\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lbv411XjJjkP"
   },
   "source": [
    "#### AutoML constants\n",
    "\n",
    "Setup up the following constants for AutoML:\n",
    "\n",
    "- `PARENT`: The AutoML location root path for dataset, model and endpoint resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rG1L9BEzxXiR"
   },
   "outputs": [],
   "source": [
    "# AutoML location root path for your dataset, model and endpoint resources\n",
    "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rw-wcM5qU7h0"
   },
   "source": [
    "## Clients\n",
    "\n",
    "The AutoML SDK works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the server (AutoML).\n",
    "\n",
    "You will use several clients in this tutorial, so set them all up upfront.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "c3sR3OxmxXiR",
    "outputId": "5bc213e7-2df8-4aca-dbf8-f8d267e98d52"
   },
   "outputs": [],
   "source": [
    "def automl_client():\n",
    "    return automl.AutoMlClient()\n",
    "\n",
    "\n",
    "def operations_client():\n",
    "    return automl.AutoMlClient()._transport.operations_client\n",
    "\n",
    "\n",
    "def prediction_client():\n",
    "    return automl.PredictionServiceClient()\n",
    "\n",
    "\n",
    "def tables_client():\n",
    "    return automl.TablesClient(project=PROJECT_ID, region=REGION)\n",
    "\n",
    "\n",
    "clients = {}\n",
    "clients[\"automl\"] = automl_client()\n",
    "clients[\"operations\"] = operations_client()\n",
    "clients[\"prediction\"] = prediction_client()\n",
    "clients[\"tables\"] = tables_client()\n",
    "\n",
    "\n",
    "for client in clients.items():\n",
    "    print(client)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "import_file:flowers,csv,icn"
   },
   "outputs": [],
   "source": [
    "IMPORT_FILE = 'gs://cloud-ml-tables-data/bank-marketing.csv'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KHJnNxI5xXiS",
    "outputId": "7714d066-2301-4b47-d155-d3c5d454def7"
   },
   "outputs": [],
   "source": [
    "! gsutil cat $IMPORT_FILE | head -n 10 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_create:migration,new,request"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "Age,Job,MaritalStatus,Education,Default,Balance,Housing,Loan,Contact,Day,Month,Duration,Campaign,PDays,Previous,POutcome,Deposit\n",
    "58,management,married,tertiary,no,2143,yes,no,unknown,5,may,261,1,-1,0,unknown,1\n",
    "44,technician,single,secondary,no,29,yes,no,unknown,5,may,151,1,-1,0,unknown,1\n",
    "33,entrepreneur,married,secondary,no,2,yes,yes,unknown,5,may,76,1,-1,0,unknown,1\n",
    "47,blue-collar,married,unknown,no,1506,yes,no,unknown,5,may,92,1,-1,0,unknown,1\n",
    "33,unknown,single,unknown,no,1,no,no,unknown,5,may,198,1,-1,0,unknown,1\n",
    "35,management,married,tertiary,no,231,yes,no,unknown,5,may,139,1,-1,0,unknown,1\n",
    "28,management,single,tertiary,no,447,yes,yes,unknown,5,may,217,1,-1,0,unknown,1\n",
    "42,entrepreneur,divorced,tertiary,yes,2,yes,no,unknown,5,may,380,1,-1,0,unknown,1\n",
    "58,retired,married,primary,no,121,yes,no,unknown,5,may,50,1,-1,0,unknown,1\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_create_dataset:migration"
   },
   "source": [
    "## Create a dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eNQ_vjVFU7h8"
   },
   "source": [
    "### Prepare data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_datasets_create:migration,old"
   },
   "source": [
    "### [projects.locations.datasets.create](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.datasets/create)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ozaMOwdNJjkT"
   },
   "source": [
    "#### Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Rm2MLKeBU7iD",
    "outputId": "2100960c-bfa5-42ba-a27c-d2047d106e10"
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(\n",
    "    automl.CreateDatasetRequest(\n",
    "        dataset=automl.Dataset(\n",
    "            display_name=\"bank_\" + TIMESTAMP\n",
    "        )\n",
    "    ).__dict__[\"_pb\"])\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"dataset\": {\n",
    "    \"displayName\": \"bank_20210228161105\"\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vHkuSZxkJjkT"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SucGG7NhU7iE"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].create_dataset(\n",
    "    dataset_display_name=\"bank_\" + TIMESTAMP\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FLqkZMD2JjkT"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_XZiIFP4U7iT",
    "outputId": "f601b5e0-5f3d-438d-d942-e17e8f7360b8"
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(request.__dict__[\"_pb\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/datasets/TBL3707245895360708608\",\n",
    "  \"displayName\": \"bank_20210228161105\",\n",
    "  \"createTime\": \"2021-02-28T16:11:11.629918Z\",\n",
    "  \"etag\": \"AB3BwFrEEPSvg_CecNSkJ7f0ROXZnZxzgAD5hxVGnRz7NlK5a_WrkIfS0tGbkJTeYFpQ\",\n",
    "  \"tablesDatasetMetadata\": {\n",
    "    \"statsUpdateTime\": \"1970-01-01T00:00:00Z\"\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The full unique ID for the dataset\n",
    "dataset_id = request.name\n",
    "# The short numeric ID for the dataset\n",
    "dataset_short_id = dataset_id.split('/')[-1]\n",
    "\n",
    "print(dataset_id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_datasets_importdata:migration,old"
   },
   "source": [
    "### [projects.locations.datasets.importData](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.datasets/importData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nacQ0lvTU7iU"
   },
   "source": [
    "#### Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6F894YodU7iZ",
    "outputId": "25becc2c-04f4-4dc8-e8d4-3c0d1bae1122"
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(\n",
    "    automl.ImportDataRequest(\n",
    "        name=dataset_id,\n",
    "        input_config={\"gcs_source\": {\"input_uris\": [IMPORT_FILE]}}\n",
    "    ).__dict__[\"_pb\"])\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/datasets/TBL3707245895360708608\",\n",
    "  \"inputConfig\": {\n",
    "    \"gcsSource\": {\n",
    "      \"inputUris\": [\n",
    "        \"gs://cloud-ml-tables-data/bank-marketing.csv\"\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NEE3oQPOU7ia"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TitFNA4FU7ia"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].import_data(\n",
    "    dataset_name=dataset_id,\n",
    "    gcs_input_uris=[IMPORT_FILE]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BfEBJzZ_U7ib"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Z5_f9IT5U7ib",
    "outputId": "c061c7b5-9da8-47a2-b084-6aee028e428d"
   },
   "outputs": [],
   "source": [
    "result = request.result()\n",
    "\n",
    "print(MessageToJson(result))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-zzom8TCkTzC"
   },
   "source": [
    "*Example output*:\r\n",
    "```\r\n",
    "{}\r\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zYDxpb87U7ig"
   },
   "source": [
    "### [projects.locations.datasets.patch](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.datasets/patch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DS-tkN-lU7ig"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TRGKYwFWU7ih"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].set_target_column(\n",
    "    dataset_name=dataset_id,\n",
    "    column_spec_display_name='Deposit'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LoBmmTiHU7ii"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Kfji8sSjU7im",
    "outputId": "99544fd6-95d5-43fe-de72-67ec3a921f58"
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(request.__dict__[\"_pb\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/datasets/TBL3707245895360708608\",\n",
    "  \"displayName\": \"bank_20210228161105\",\n",
    "  \"createTime\": \"2021-02-28T16:11:11.629918Z\",\n",
    "  \"etag\": \"AB3BwFoGe9VDggahb_dNWX28GjKcqW9XDUytMYvTDIoFWTv2pbmfLrpMB2kge33oW2MY\",\n",
    "  \"exampleCount\": 45211,\n",
    "  \"tablesDatasetMetadata\": {\n",
    "    \"primaryTableSpecId\": \"7767359156935196672\",\n",
    "    \"targetColumnSpecId\": \"2361404080544284672\",\n",
    "    \"statsUpdateTime\": \"2021-02-28T16:11:54.368477Z\"\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uFP7B00cU7in"
   },
   "source": [
    "## Train a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_models_create:migration,old"
   },
   "source": [
    "### [projects.locations.models.create](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.models/create)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wX-bPrRTU7io"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dYN44cllU7io"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].create_model(\n",
    "    model_display_name=\"bank_\" + TIMESTAMP,\n",
    "    dataset_name=dataset_id,\n",
    "    train_budget_milli_node_hours=1*1000\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GJu9d2CFU7it"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_4frkgiOU7it",
    "outputId": "737dfa45-7623-4d16-cec0-4b77fd781c8c"
   },
   "outputs": [],
   "source": [
    "result = request.result()\n",
    "\n",
    "print(MessageToJson(result.__dict__[\"_pb\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264\"\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The full unique ID for the training pipeline\n",
    "model_id = result.name\n",
    "# The short numeric ID for the training pipeline\n",
    "model_short_id = model_id.split('/')[-1]\n",
    "\n",
    "print(model_id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_evaluate_the_model:migration"
   },
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yfufMwAEJjkX"
   },
   "source": [
    "### [projects.locations.models.modelEvaluations.list](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.models.modelEvaluations/list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SjYvkggyU7iy"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wI1d4wFsU7iz"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].list_model_evaluations(\n",
    "    model_name=model_id\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jFRv00YOU7iz"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DwVP9IIiU7i1",
    "outputId": "2b212216-331b-40b3-fecb-8e497ca1e52b",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "evaluations_list = [\n",
    "    json.loads(MessageToJson(me.__dict__[\"_pb\"])) \n",
    "    for me in request.model_evaluation\n",
    "]\n",
    "\n",
    "print(json.dumps(evaluations_list, indent=2))\n",
    "# The  evaluation slice\n",
    "evaluation_slice = request.model_evaluation[0].name\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "[\n",
    "  {\n",
    "    \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/124023939996839080\",\n",
    "    \"annotationSpecId\": \"not available\",\n",
    "    \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "    \"evaluatedExampleCount\": 4639,\n",
    "    \"classificationEvaluationMetrics\": {\n",
    "      \"auPrc\": 0.89604473,\n",
    "      \"confidenceMetricsEntry\": [\n",
    "        {\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.009267787,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.08771089,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.18867286,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.24711765,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.29667678,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.33910435,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.3722213,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.41141066,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.44597256,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.48088387,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51082355,\n",
    "          \"recall\": 0.9006251,\n",
    "          \"precision\": 0.90688086,\n",
    "          \"f1Score\": 0.90374213,\n",
    "          \"falsePositiveRate\": 0.09247683,\n",
    "          \"truePositiveCount\": \"4178\",\n",
    "          \"falsePositiveCount\": \"429\",\n",
    "          \"falseNegativeCount\": \"461\",\n",
    "          \"trueNegativeCount\": \"4210\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5395309,\n",
    "          \"recall\": 0.8902781,\n",
    "          \"precision\": 0.9145261,\n",
    "          \"f1Score\": 0.9022392,\n",
    "          \"falsePositiveRate\": 0.083207585,\n",
    "          \"truePositiveCount\": \"4130\",\n",
    "          \"falsePositiveCount\": \"386\",\n",
    "          \"falseNegativeCount\": \"509\",\n",
    "          \"trueNegativeCount\": \"4253\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.569127,\n",
    "          \"recall\": 0.88014656,\n",
    "          \"precision\": 0.92083895,\n",
    "          \"f1Score\": 0.90003306,\n",
    "          \"falsePositiveRate\": 0.07566286,\n",
    "          \"truePositiveCount\": \"4083\",\n",
    "          \"falsePositiveCount\": \"351\",\n",
    "          \"falseNegativeCount\": \"556\",\n",
    "          \"trueNegativeCount\": \"4288\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.59819055,\n",
    "          \"recall\": 0.8700151,\n",
    "          \"precision\": 0.9284564,\n",
    "          \"f1Score\": 0.8982862,\n",
    "          \"falsePositiveRate\": 0.06704031,\n",
    "          \"truePositiveCount\": \"4036\",\n",
    "          \"falsePositiveCount\": \"311\",\n",
    "          \"falseNegativeCount\": \"603\",\n",
    "          \"trueNegativeCount\": \"4328\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6267493,\n",
    "          \"recall\": 0.86009914,\n",
    "          \"precision\": 0.9350832,\n",
    "          \"f1Score\": 0.8960252,\n",
    "          \"falsePositiveRate\": 0.059711143,\n",
    "          \"truePositiveCount\": \"3990\",\n",
    "          \"falsePositiveCount\": \"277\",\n",
    "          \"falseNegativeCount\": \"649\",\n",
    "          \"trueNegativeCount\": \"4362\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6483357,\n",
    "          \"recall\": 0.84996766,\n",
    "          \"precision\": 0.9401526,\n",
    "          \"f1Score\": 0.8927884,\n",
    "          \"falsePositiveRate\": 0.05410649,\n",
    "          \"truePositiveCount\": \"3943\",\n",
    "          \"falsePositiveCount\": \"251\",\n",
    "          \"falseNegativeCount\": \"696\",\n",
    "          \"trueNegativeCount\": \"4388\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.68742466,\n",
    "          \"recall\": 0.8404829,\n",
    "          \"precision\": 0.95051193,\n",
    "          \"f1Score\": 0.8921176,\n",
    "          \"falsePositiveRate\": 0.04375943,\n",
    "          \"truePositiveCount\": \"3899\",\n",
    "          \"falsePositiveCount\": \"203\",\n",
    "          \"falseNegativeCount\": \"740\",\n",
    "          \"trueNegativeCount\": \"4436\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7178758,\n",
    "          \"recall\": 0.83035135,\n",
    "          \"precision\": 0.95630586,\n",
    "          \"f1Score\": 0.8888889,\n",
    "          \"falsePositiveRate\": 0.03793921,\n",
    "          \"truePositiveCount\": \"3852\",\n",
    "          \"falsePositiveCount\": \"176\",\n",
    "          \"falseNegativeCount\": \"787\",\n",
    "          \"trueNegativeCount\": \"4463\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7454512,\n",
    "          \"recall\": 0.8200043,\n",
    "          \"precision\": 0.9620637,\n",
    "          \"f1Score\": 0.8853718,\n",
    "          \"falsePositiveRate\": 0.032334555,\n",
    "          \"truePositiveCount\": \"3804\",\n",
    "          \"falsePositiveCount\": \"150\",\n",
    "          \"falseNegativeCount\": \"835\",\n",
    "          \"trueNegativeCount\": \"4489\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7702336,\n",
    "          \"recall\": 0.8105195,\n",
    "          \"precision\": 0.9685729,\n",
    "          \"f1Score\": 0.8825255,\n",
    "          \"falsePositiveRate\": 0.02629877,\n",
    "          \"truePositiveCount\": \"3760\",\n",
    "          \"falsePositiveCount\": \"122\",\n",
    "          \"falseNegativeCount\": \"879\",\n",
    "          \"trueNegativeCount\": \"4517\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.79181457,\n",
    "          \"recall\": 0.80038804,\n",
    "          \"precision\": 0.972244,\n",
    "          \"f1Score\": 0.87798536,\n",
    "          \"falsePositiveRate\": 0.022849752,\n",
    "          \"truePositiveCount\": \"3713\",\n",
    "          \"falsePositiveCount\": \"106\",\n",
    "          \"falseNegativeCount\": \"926\",\n",
    "          \"trueNegativeCount\": \"4533\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.80905926,\n",
    "          \"recall\": 0.7902565,\n",
    "          \"precision\": 0.9755189,\n",
    "          \"f1Score\": 0.873169,\n",
    "          \"falsePositiveRate\": 0.01983186,\n",
    "          \"truePositiveCount\": \"3666\",\n",
    "          \"falsePositiveCount\": \"92\",\n",
    "          \"falseNegativeCount\": \"973\",\n",
    "          \"trueNegativeCount\": \"4547\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.82475233,\n",
    "          \"recall\": 0.7803406,\n",
    "          \"precision\": 0.97574127,\n",
    "          \"f1Score\": 0.86716974,\n",
    "          \"falsePositiveRate\": 0.019400733,\n",
    "          \"truePositiveCount\": \"3620\",\n",
    "          \"falsePositiveCount\": \"90\",\n",
    "          \"falseNegativeCount\": \"1019\",\n",
    "          \"trueNegativeCount\": \"4549\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.84347624,\n",
    "          \"recall\": 0.7702091,\n",
    "          \"precision\": 0.97863597,\n",
    "          \"f1Score\": 0.86200243,\n",
    "          \"falsePositiveRate\": 0.01681397,\n",
    "          \"truePositiveCount\": \"3573\",\n",
    "          \"falsePositiveCount\": \"78\",\n",
    "          \"falseNegativeCount\": \"1066\",\n",
    "          \"trueNegativeCount\": \"4561\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8612723,\n",
    "          \"recall\": 0.7602932,\n",
    "          \"precision\": 0.9799944,\n",
    "          \"f1Score\": 0.8562758,\n",
    "          \"falsePositiveRate\": 0.015520587,\n",
    "          \"truePositiveCount\": \"3527\",\n",
    "          \"falsePositiveCount\": \"72\",\n",
    "          \"falseNegativeCount\": \"1112\",\n",
    "          \"trueNegativeCount\": \"4567\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.87450475,\n",
    "          \"recall\": 0.75016165,\n",
    "          \"precision\": 0.9811108,\n",
    "          \"f1Score\": 0.8502321,\n",
    "          \"falsePositiveRate\": 0.014442768,\n",
    "          \"truePositiveCount\": \"3480\",\n",
    "          \"falsePositiveCount\": \"67\",\n",
    "          \"falseNegativeCount\": \"1159\",\n",
    "          \"trueNegativeCount\": \"4572\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8876167,\n",
    "          \"recall\": 0.74003017,\n",
    "          \"precision\": 0.9825415,\n",
    "          \"f1Score\": 0.8442149,\n",
    "          \"falsePositiveRate\": 0.013149385,\n",
    "          \"truePositiveCount\": \"3433\",\n",
    "          \"falsePositiveCount\": \"61\",\n",
    "          \"falseNegativeCount\": \"1206\",\n",
    "          \"trueNegativeCount\": \"4578\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.89729506,\n",
    "          \"recall\": 0.7303298,\n",
    "          \"precision\": 0.98402554,\n",
    "          \"f1Score\": 0.8384063,\n",
    "          \"falsePositiveRate\": 0.011856004,\n",
    "          \"truePositiveCount\": \"3388\",\n",
    "          \"falsePositiveCount\": \"55\",\n",
    "          \"falseNegativeCount\": \"1251\",\n",
    "          \"trueNegativeCount\": \"4584\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9069939,\n",
    "          \"recall\": 0.72041386,\n",
    "          \"precision\": 0.9858407,\n",
    "          \"f1Score\": 0.8324823,\n",
    "          \"falsePositiveRate\": 0.010347057,\n",
    "          \"truePositiveCount\": \"3342\",\n",
    "          \"falsePositiveCount\": \"48\",\n",
    "          \"falseNegativeCount\": \"1297\",\n",
    "          \"trueNegativeCount\": \"4591\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.91544837,\n",
    "          \"recall\": 0.7102824,\n",
    "          \"precision\": 0.987118,\n",
    "          \"f1Score\": 0.8261251,\n",
    "          \"falsePositiveRate\": 0.009269239,\n",
    "          \"truePositiveCount\": \"3295\",\n",
    "          \"falsePositiveCount\": \"43\",\n",
    "          \"falseNegativeCount\": \"1344\",\n",
    "          \"trueNegativeCount\": \"4596\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9246296,\n",
    "          \"recall\": 0.700582,\n",
    "          \"precision\": 0.9896468,\n",
    "          \"f1Score\": 0.8203963,\n",
    "          \"falsePositiveRate\": 0.007329166,\n",
    "          \"truePositiveCount\": \"3250\",\n",
    "          \"falsePositiveCount\": \"34\",\n",
    "          \"falseNegativeCount\": \"1389\",\n",
    "          \"trueNegativeCount\": \"4605\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.932719,\n",
    "          \"recall\": 0.6900194,\n",
    "          \"precision\": 0.99071497,\n",
    "          \"f1Score\": 0.8134689,\n",
    "          \"falsePositiveRate\": 0.006466911,\n",
    "          \"truePositiveCount\": \"3201\",\n",
    "          \"falsePositiveCount\": \"30\",\n",
    "          \"falseNegativeCount\": \"1438\",\n",
    "          \"trueNegativeCount\": \"4609\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.93940556,\n",
    "          \"recall\": 0.6807502,\n",
    "          \"precision\": 0.9921458,\n",
    "          \"f1Score\": 0.80746615,\n",
    "          \"falsePositiveRate\": 0.0053890925,\n",
    "          \"truePositiveCount\": \"3158\",\n",
    "          \"falsePositiveCount\": \"25\",\n",
    "          \"falseNegativeCount\": \"1481\",\n",
    "          \"trueNegativeCount\": \"4614\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9436984,\n",
    "          \"recall\": 0.6704031,\n",
    "          \"precision\": 0.99234205,\n",
    "          \"f1Score\": 0.8002058,\n",
    "          \"falsePositiveRate\": 0.0051735286,\n",
    "          \"truePositiveCount\": \"3110\",\n",
    "          \"falsePositiveCount\": \"24\",\n",
    "          \"falseNegativeCount\": \"1529\",\n",
    "          \"trueNegativeCount\": \"4615\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9486017,\n",
    "          \"recall\": 0.6602716,\n",
    "          \"precision\": 0.992547,\n",
    "          \"f1Score\": 0.7930097,\n",
    "          \"falsePositiveRate\": 0.004957965,\n",
    "          \"truePositiveCount\": \"3063\",\n",
    "          \"falsePositiveCount\": \"23\",\n",
    "          \"falseNegativeCount\": \"1576\",\n",
    "          \"trueNegativeCount\": \"4616\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95302504,\n",
    "          \"recall\": 0.6501401,\n",
    "          \"precision\": 0.9937397,\n",
    "          \"f1Score\": 0.78603077,\n",
    "          \"falsePositiveRate\": 0.0040957103,\n",
    "          \"truePositiveCount\": \"3016\",\n",
    "          \"falsePositiveCount\": \"19\",\n",
    "          \"falseNegativeCount\": \"1623\",\n",
    "          \"trueNegativeCount\": \"4620\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95584136,\n",
    "          \"recall\": 0.6400086,\n",
    "          \"precision\": 0.99530673,\n",
    "          \"f1Score\": 0.7790606,\n",
    "          \"falsePositiveRate\": 0.003017892,\n",
    "          \"truePositiveCount\": \"2969\",\n",
    "          \"falsePositiveCount\": \"14\",\n",
    "          \"falseNegativeCount\": \"1670\",\n",
    "          \"trueNegativeCount\": \"4625\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95881057,\n",
    "          \"recall\": 0.6300927,\n",
    "          \"precision\": 0.9955722,\n",
    "          \"f1Score\": 0.7717492,\n",
    "          \"falsePositiveRate\": 0.002802328,\n",
    "          \"truePositiveCount\": \"2923\",\n",
    "          \"falsePositiveCount\": \"13\",\n",
    "          \"falseNegativeCount\": \"1716\",\n",
    "          \"trueNegativeCount\": \"4626\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96166563,\n",
    "          \"recall\": 0.62082344,\n",
    "          \"precision\": 0.9961951,\n",
    "          \"f1Score\": 0.76494026,\n",
    "          \"falsePositiveRate\": 0.0023712006,\n",
    "          \"truePositiveCount\": \"2880\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1759\",\n",
    "          \"trueNegativeCount\": \"4628\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96420497,\n",
    "          \"recall\": 0.61026084,\n",
    "          \"precision\": 0.9961295,\n",
    "          \"f1Score\": 0.75685066,\n",
    "          \"falsePositiveRate\": 0.0023712006,\n",
    "          \"truePositiveCount\": \"2831\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1808\",\n",
    "          \"trueNegativeCount\": \"4628\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96650803,\n",
    "          \"recall\": 0.5999138,\n",
    "          \"precision\": 0.996063,\n",
    "          \"f1Score\": 0.7488228,\n",
    "          \"falsePositiveRate\": 0.0023712006,\n",
    "          \"truePositiveCount\": \"2783\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1856\",\n",
    "          \"trueNegativeCount\": \"4628\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96897614,\n",
    "          \"recall\": 0.59042895,\n",
    "          \"precision\": 0.9963623,\n",
    "          \"f1Score\": 0.74147266,\n",
    "          \"falsePositiveRate\": 0.002155637,\n",
    "          \"truePositiveCount\": \"2739\",\n",
    "          \"falsePositiveCount\": \"10\",\n",
    "          \"falseNegativeCount\": \"1900\",\n",
    "          \"trueNegativeCount\": \"4629\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97092766,\n",
    "          \"recall\": 0.5807286,\n",
    "          \"precision\": 0.99667037,\n",
    "          \"f1Score\": 0.73385996,\n",
    "          \"falsePositiveRate\": 0.0019400733,\n",
    "          \"truePositiveCount\": \"2694\",\n",
    "          \"falsePositiveCount\": \"9\",\n",
    "          \"falseNegativeCount\": \"1945\",\n",
    "          \"trueNegativeCount\": \"4630\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9727514,\n",
    "          \"recall\": 0.5699504,\n",
    "          \"precision\": 0.9969834,\n",
    "          \"f1Score\": 0.7252777,\n",
    "          \"falsePositiveRate\": 0.0017245096,\n",
    "          \"truePositiveCount\": \"2644\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"1995\",\n",
    "          \"trueNegativeCount\": \"4631\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9747563,\n",
    "          \"recall\": 0.56046563,\n",
    "          \"precision\": 0.99731493,\n",
    "          \"f1Score\": 0.7176373,\n",
    "          \"falsePositiveRate\": 0.001508946,\n",
    "          \"truePositiveCount\": \"2600\",\n",
    "          \"falsePositiveCount\": \"7\",\n",
    "          \"falseNegativeCount\": \"2039\",\n",
    "          \"trueNegativeCount\": \"4632\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9766838,\n",
    "          \"recall\": 0.549903,\n",
    "          \"precision\": 0.9976535,\n",
    "          \"f1Score\": 0.709005,\n",
    "          \"falsePositiveRate\": 0.0012933821,\n",
    "          \"truePositiveCount\": \"2551\",\n",
    "          \"falsePositiveCount\": \"6\",\n",
    "          \"falseNegativeCount\": \"2088\",\n",
    "          \"trueNegativeCount\": \"4633\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97776866,\n",
    "          \"recall\": 0.54063374,\n",
    "          \"precision\": 0.99801034,\n",
    "          \"f1Score\": 0.7013423,\n",
    "          \"falsePositiveRate\": 0.0010778185,\n",
    "          \"truePositiveCount\": \"2508\",\n",
    "          \"falsePositiveCount\": \"5\",\n",
    "          \"falseNegativeCount\": \"2131\",\n",
    "          \"trueNegativeCount\": \"4634\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9792647,\n",
    "          \"recall\": 0.53050226,\n",
    "          \"precision\": 0.9979724,\n",
    "          \"f1Score\": 0.6927516,\n",
    "          \"falsePositiveRate\": 0.0010778185,\n",
    "          \"truePositiveCount\": \"2461\",\n",
    "          \"falsePositiveCount\": \"5\",\n",
    "          \"falseNegativeCount\": \"2178\",\n",
    "          \"trueNegativeCount\": \"4634\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9804138,\n",
    "          \"recall\": 0.5203708,\n",
    "          \"precision\": 0.99834573,\n",
    "          \"f1Score\": 0.6841434,\n",
    "          \"falsePositiveRate\": 0.0008622548,\n",
    "          \"truePositiveCount\": \"2414\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"2225\",\n",
    "          \"trueNegativeCount\": \"4635\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9814273,\n",
    "          \"recall\": 0.51045483,\n",
    "          \"precision\": 0.9991561,\n",
    "          \"f1Score\": 0.6757027,\n",
    "          \"falsePositiveRate\": 0.0004311274,\n",
    "          \"truePositiveCount\": \"2368\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"2271\",\n",
    "          \"trueNegativeCount\": \"4637\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98243207,\n",
    "          \"recall\": 0.5005389,\n",
    "          \"precision\": 0.9991394,\n",
    "          \"f1Score\": 0.6669539,\n",
    "          \"falsePositiveRate\": 0.0004311274,\n",
    "          \"truePositiveCount\": \"2322\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"2317\",\n",
    "          \"trueNegativeCount\": \"4637\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9835001,\n",
    "          \"recall\": 0.49019185,\n",
    "          \"precision\": 0.9995604,\n",
    "          \"f1Score\": 0.6577958,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2274\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2365\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9844133,\n",
    "          \"recall\": 0.4804915,\n",
    "          \"precision\": 0.9995516,\n",
    "          \"f1Score\": 0.6490028,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2229\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2410\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98528725,\n",
    "          \"recall\": 0.47079113,\n",
    "          \"precision\": 0.99954236,\n",
    "          \"f1Score\": 0.6400938,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2184\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2455\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98630506,\n",
    "          \"recall\": 0.46001294,\n",
    "          \"precision\": 0.9995316,\n",
    "          \"f1Score\": 0.6300561,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2134\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2505\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9871925,\n",
    "          \"recall\": 0.44988143,\n",
    "          \"precision\": 0.9995211,\n",
    "          \"f1Score\": 0.6204846,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2087\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2552\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9877509,\n",
    "          \"recall\": 0.44082776,\n",
    "          \"precision\": 0.99951124,\n",
    "          \"f1Score\": 0.6118175,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2045\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2594\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9885219,\n",
    "          \"recall\": 0.43048072,\n",
    "          \"precision\": 0.9994995,\n",
    "          \"f1Score\": 0.6017779,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1997\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2642\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9892192,\n",
    "          \"recall\": 0.4199181,\n",
    "          \"precision\": 0.9994869,\n",
    "          \"f1Score\": 0.5913783,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1948\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2691\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9897657,\n",
    "          \"recall\": 0.41000214,\n",
    "          \"precision\": 0.9994745,\n",
    "          \"f1Score\": 0.5814735,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1902\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2737\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9902156,\n",
    "          \"recall\": 0.40051734,\n",
    "          \"precision\": 0.99946207,\n",
    "          \"f1Score\": 0.57186824,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1858\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2781\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.990671,\n",
    "          \"recall\": 0.3901703,\n",
    "          \"precision\": 0.9994478,\n",
    "          \"f1Score\": 0.5612403,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1810\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2829\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99106044,\n",
    "          \"recall\": 0.38025436,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5509917,\n",
    "          \"truePositiveCount\": \"1764\",\n",
    "          \"falseNegativeCount\": \"2875\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99142385,\n",
    "          \"recall\": 0.36990732,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5400472,\n",
    "          \"truePositiveCount\": \"1716\",\n",
    "          \"falseNegativeCount\": \"2923\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.991821,\n",
    "          \"recall\": 0.35999137,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.52940243,\n",
    "          \"truePositiveCount\": \"1670\",\n",
    "          \"falseNegativeCount\": \"2969\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99219143,\n",
    "          \"recall\": 0.35007545,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5186013,\n",
    "          \"truePositiveCount\": \"1624\",\n",
    "          \"falseNegativeCount\": \"3015\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99265563,\n",
    "          \"recall\": 0.3401595,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.50764036,\n",
    "          \"truePositiveCount\": \"1578\",\n",
    "          \"falseNegativeCount\": \"3061\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9930375,\n",
    "          \"recall\": 0.33045915,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.49675956,\n",
    "          \"truePositiveCount\": \"1533\",\n",
    "          \"falseNegativeCount\": \"3106\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9932301,\n",
    "          \"recall\": 0.32054323,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.48547176,\n",
    "          \"truePositiveCount\": \"1487\",\n",
    "          \"falseNegativeCount\": \"3152\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99349546,\n",
    "          \"recall\": 0.31041172,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.47376212,\n",
    "          \"truePositiveCount\": \"1440\",\n",
    "          \"falseNegativeCount\": \"3199\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.993766,\n",
    "          \"recall\": 0.30006468,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.461615,\n",
    "          \"truePositiveCount\": \"1392\",\n",
    "          \"falseNegativeCount\": \"3247\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9940096,\n",
    "          \"recall\": 0.29014874,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.44979113,\n",
    "          \"truePositiveCount\": \"1346\",\n",
    "          \"falseNegativeCount\": \"3293\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9942971,\n",
    "          \"recall\": 0.27980167,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.4372579,\n",
    "          \"truePositiveCount\": \"1298\",\n",
    "          \"falseNegativeCount\": \"3341\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9945464,\n",
    "          \"recall\": 0.26988575,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.42505518,\n",
    "          \"truePositiveCount\": \"1252\",\n",
    "          \"falseNegativeCount\": \"3387\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.994734,\n",
    "          \"recall\": 0.2601854,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.41293192,\n",
    "          \"truePositiveCount\": \"1207\",\n",
    "          \"falseNegativeCount\": \"3432\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9949861,\n",
    "          \"recall\": 0.25026944,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.40034482,\n",
    "          \"truePositiveCount\": \"1161\",\n",
    "          \"falseNegativeCount\": \"3478\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99523896,\n",
    "          \"recall\": 0.2399224,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.38699582,\n",
    "          \"truePositiveCount\": \"1113\",\n",
    "          \"falseNegativeCount\": \"3526\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99545944,\n",
    "          \"recall\": 0.23043759,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.37456203,\n",
    "          \"truePositiveCount\": \"1069\",\n",
    "          \"falseNegativeCount\": \"3570\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9956529,\n",
    "          \"recall\": 0.22009054,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.36077738,\n",
    "          \"truePositiveCount\": \"1021\",\n",
    "          \"falseNegativeCount\": \"3618\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9958653,\n",
    "          \"recall\": 0.21039017,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.34764025,\n",
    "          \"truePositiveCount\": \"976\",\n",
    "          \"falseNegativeCount\": \"3663\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99604857,\n",
    "          \"recall\": 0.20025867,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.33369252,\n",
    "          \"truePositiveCount\": \"929\",\n",
    "          \"falseNegativeCount\": \"3710\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99617386,\n",
    "          \"recall\": 0.19012718,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.31950733,\n",
    "          \"truePositiveCount\": \"882\",\n",
    "          \"falseNegativeCount\": \"3757\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9963188,\n",
    "          \"recall\": 0.18042682,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.3056976,\n",
    "          \"truePositiveCount\": \"837\",\n",
    "          \"falseNegativeCount\": \"3802\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99644965,\n",
    "          \"recall\": 0.17029533,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.29102966,\n",
    "          \"truePositiveCount\": \"790\",\n",
    "          \"falseNegativeCount\": \"3849\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99657893,\n",
    "          \"recall\": 0.16059496,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.27674592,\n",
    "          \"truePositiveCount\": \"745\",\n",
    "          \"falseNegativeCount\": \"3894\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99672806,\n",
    "          \"recall\": 0.1502479,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.2612444,\n",
    "          \"truePositiveCount\": \"697\",\n",
    "          \"falseNegativeCount\": \"3942\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9968369,\n",
    "          \"recall\": 0.14033197,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.24612476,\n",
    "          \"truePositiveCount\": \"651\",\n",
    "          \"falseNegativeCount\": \"3988\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9969655,\n",
    "          \"recall\": 0.13041604,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.23073989,\n",
    "          \"truePositiveCount\": \"605\",\n",
    "          \"falseNegativeCount\": \"4034\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9970963,\n",
    "          \"recall\": 0.12028454,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.21473928,\n",
    "          \"truePositiveCount\": \"558\",\n",
    "          \"falseNegativeCount\": \"4081\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9972023,\n",
    "          \"recall\": 0.110799745,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.19949543,\n",
    "          \"truePositiveCount\": \"514\",\n",
    "          \"falseNegativeCount\": \"4125\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9973033,\n",
    "          \"recall\": 0.100668244,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.18292205,\n",
    "          \"truePositiveCount\": \"467\",\n",
    "          \"falseNegativeCount\": \"4172\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99741936,\n",
    "          \"recall\": 0.09053675,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.16604072,\n",
    "          \"truePositiveCount\": \"420\",\n",
    "          \"falseNegativeCount\": \"4219\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9974875,\n",
    "          \"recall\": 0.080620825,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.14921205,\n",
    "          \"truePositiveCount\": \"374\",\n",
    "          \"falseNegativeCount\": \"4265\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9976044,\n",
    "          \"recall\": 0.070273764,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.13131924,\n",
    "          \"truePositiveCount\": \"326\",\n",
    "          \"falseNegativeCount\": \"4313\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99769753,\n",
    "          \"recall\": 0.06014227,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.11346076,\n",
    "          \"truePositiveCount\": \"279\",\n",
    "          \"falseNegativeCount\": \"4360\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99778664,\n",
    "          \"recall\": 0.049795214,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.09486653,\n",
    "          \"truePositiveCount\": \"231\",\n",
    "          \"falseNegativeCount\": \"4408\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9978742,\n",
    "          \"recall\": 0.040525977,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.07789517,\n",
    "          \"truePositiveCount\": \"188\",\n",
    "          \"falseNegativeCount\": \"4451\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.997992,\n",
    "          \"recall\": 0.030394481,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.058995817,\n",
    "          \"truePositiveCount\": \"141\",\n",
    "          \"falseNegativeCount\": \"4498\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9980989,\n",
    "          \"recall\": 0.020262988,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.03972111,\n",
    "          \"truePositiveCount\": \"94\",\n",
    "          \"falseNegativeCount\": \"4545\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99821925,\n",
    "          \"recall\": 0.010347057,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.020482184,\n",
    "          \"truePositiveCount\": \"48\",\n",
    "          \"falseNegativeCount\": \"4591\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9987167,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4639\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 1.0,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4639\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 1\n",
    "        }\n",
    "      ],\n",
    "      \"confusionMatrix\": {\n",
    "        \"row\": [\n",
    "          {\n",
    "            \"exampleCount\": [\n",
    "              3941,\n",
    "              147\n",
    "            ]\n",
    "          },\n",
    "          {\n",
    "            \"exampleCount\": [\n",
    "              293,\n",
    "              258\n",
    "            ]\n",
    "          }\n",
    "        ],\n",
    "        \"displayName\": [\n",
    "          \"1\",\n",
    "          \"2\"\n",
    "        ]\n",
    "      },\n",
    "      \"auRoc\": 0.94022393,\n",
    "      \"logLoss\": 0.20077285\n",
    "    }\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/542606150386485790\",\n",
    "    \"annotationSpecId\": \"not available\",\n",
    "    \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "    \"evaluatedExampleCount\": 4088,\n",
    "    \"classificationEvaluationMetrics\": {\n",
    "      \"auPrc\": 0.9907691,\n",
    "      \"confidenceMetricsEntry\": [\n",
    "        {\n",
    "          \"recall\": 1.0,\n",
    "          \"precision\": 0.8812244,\n",
    "          \"f1Score\": 0.9368626,\n",
    "          \"falsePositiveRate\": 1.0,\n",
    "          \"truePositiveCount\": \"4088\",\n",
    "          \"falsePositiveCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.07797722,\n",
    "          \"recall\": 0.9997554,\n",
    "          \"precision\": 0.8815789,\n",
    "          \"f1Score\": 0.9369555,\n",
    "          \"falsePositiveRate\": 0.99637026,\n",
    "          \"truePositiveCount\": \"4087\",\n",
    "          \"falsePositiveCount\": \"549\",\n",
    "          \"falseNegativeCount\": \"1\",\n",
    "          \"trueNegativeCount\": \"2\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.30998033,\n",
    "          \"recall\": 0.9899706,\n",
    "          \"precision\": 0.90254235,\n",
    "          \"f1Score\": 0.94423705,\n",
    "          \"falsePositiveRate\": 0.79310346,\n",
    "          \"truePositiveCount\": \"4047\",\n",
    "          \"falsePositiveCount\": \"437\",\n",
    "          \"falseNegativeCount\": \"41\",\n",
    "          \"trueNegativeCount\": \"114\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39478266,\n",
    "          \"recall\": 0.9801859,\n",
    "          \"precision\": 0.913797,\n",
    "          \"f1Score\": 0.9458279,\n",
    "          \"falsePositiveRate\": 0.6860254,\n",
    "          \"truePositiveCount\": \"4007\",\n",
    "          \"falsePositiveCount\": \"378\",\n",
    "          \"falseNegativeCount\": \"81\",\n",
    "          \"trueNegativeCount\": \"173\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.46149504,\n",
    "          \"recall\": 0.96991193,\n",
    "          \"precision\": 0.9246735,\n",
    "          \"f1Score\": 0.9467526,\n",
    "          \"falsePositiveRate\": 0.5862069,\n",
    "          \"truePositiveCount\": \"3965\",\n",
    "          \"falsePositiveCount\": \"323\",\n",
    "          \"falseNegativeCount\": \"123\",\n",
    "          \"trueNegativeCount\": \"228\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5,\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51386535,\n",
    "          \"recall\": 0.95988256,\n",
    "          \"precision\": 0.93361884,\n",
    "          \"f1Score\": 0.94656855,\n",
    "          \"falsePositiveRate\": 0.50635207,\n",
    "          \"truePositiveCount\": \"3924\",\n",
    "          \"falsePositiveCount\": \"279\",\n",
    "          \"falseNegativeCount\": \"164\",\n",
    "          \"trueNegativeCount\": \"272\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5659395,\n",
    "          \"recall\": 0.94985324,\n",
    "          \"precision\": 0.9406492,\n",
    "          \"f1Score\": 0.9452288,\n",
    "          \"falsePositiveRate\": 0.4446461,\n",
    "          \"truePositiveCount\": \"3883\",\n",
    "          \"falsePositiveCount\": \"245\",\n",
    "          \"falseNegativeCount\": \"205\",\n",
    "          \"trueNegativeCount\": \"306\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6215324,\n",
    "          \"recall\": 0.93982387,\n",
    "          \"precision\": 0.94817376,\n",
    "          \"f1Score\": 0.94398034,\n",
    "          \"falsePositiveRate\": 0.38112524,\n",
    "          \"truePositiveCount\": \"3842\",\n",
    "          \"falsePositiveCount\": \"210\",\n",
    "          \"falseNegativeCount\": \"246\",\n",
    "          \"trueNegativeCount\": \"341\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.65884733,\n",
    "          \"recall\": 0.93003917,\n",
    "          \"precision\": 0.9533601,\n",
    "          \"f1Score\": 0.9415552,\n",
    "          \"falsePositiveRate\": 0.33756804,\n",
    "          \"truePositiveCount\": \"3802\",\n",
    "          \"falsePositiveCount\": \"186\",\n",
    "          \"falseNegativeCount\": \"286\",\n",
    "          \"trueNegativeCount\": \"365\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7143411,\n",
    "          \"recall\": 0.9200098,\n",
    "          \"precision\": 0.9623849,\n",
    "          \"f1Score\": 0.9407204,\n",
    "          \"falsePositiveRate\": 0.26678765,\n",
    "          \"truePositiveCount\": \"3761\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"327\",\n",
    "          \"trueNegativeCount\": \"404\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7492718,\n",
    "          \"recall\": 0.9099804,\n",
    "          \"precision\": 0.96749026,\n",
    "          \"f1Score\": 0.9378545,\n",
    "          \"falsePositiveRate\": 0.22686026,\n",
    "          \"truePositiveCount\": \"3720\",\n",
    "          \"falsePositiveCount\": \"125\",\n",
    "          \"falseNegativeCount\": \"368\",\n",
    "          \"trueNegativeCount\": \"426\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7795446,\n",
    "          \"recall\": 0.9001957,\n",
    "          \"precision\": 0.97303015,\n",
    "          \"f1Score\": 0.93519694,\n",
    "          \"falsePositiveRate\": 0.18511796,\n",
    "          \"truePositiveCount\": \"3680\",\n",
    "          \"falsePositiveCount\": \"102\",\n",
    "          \"falseNegativeCount\": \"408\",\n",
    "          \"trueNegativeCount\": \"449\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7989856,\n",
    "          \"recall\": 0.8899217,\n",
    "          \"precision\": 0.97559667,\n",
    "          \"f1Score\": 0.93079185,\n",
    "          \"falsePositiveRate\": 0.16515426,\n",
    "          \"truePositiveCount\": \"3638\",\n",
    "          \"falsePositiveCount\": \"91\",\n",
    "          \"falseNegativeCount\": \"450\",\n",
    "          \"trueNegativeCount\": \"460\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.82042044,\n",
    "          \"recall\": 0.87989235,\n",
    "          \"precision\": 0.9777113,\n",
    "          \"f1Score\": 0.9262263,\n",
    "          \"falsePositiveRate\": 0.14882033,\n",
    "          \"truePositiveCount\": \"3597\",\n",
    "          \"falsePositiveCount\": \"82\",\n",
    "          \"falseNegativeCount\": \"491\",\n",
    "          \"trueNegativeCount\": \"469\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.83562684,\n",
    "          \"recall\": 0.87010765,\n",
    "          \"precision\": 0.9790807,\n",
    "          \"f1Score\": 0.92138326,\n",
    "          \"falsePositiveRate\": 0.13793103,\n",
    "          \"truePositiveCount\": \"3557\",\n",
    "          \"falsePositiveCount\": \"76\",\n",
    "          \"falseNegativeCount\": \"531\",\n",
    "          \"trueNegativeCount\": \"475\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.85593396,\n",
    "          \"recall\": 0.8600783,\n",
    "          \"precision\": 0.98102677,\n",
    "          \"f1Score\": 0.9165798,\n",
    "          \"falsePositiveRate\": 0.123411976,\n",
    "          \"truePositiveCount\": \"3516\",\n",
    "          \"falsePositiveCount\": \"68\",\n",
    "          \"falseNegativeCount\": \"572\",\n",
    "          \"trueNegativeCount\": \"483\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.86959773,\n",
    "          \"recall\": 0.8500489,\n",
    "          \"precision\": 0.9810841,\n",
    "          \"f1Score\": 0.9108781,\n",
    "          \"falsePositiveRate\": 0.1215971,\n",
    "          \"truePositiveCount\": \"3475\",\n",
    "          \"falsePositiveCount\": \"67\",\n",
    "          \"falseNegativeCount\": \"613\",\n",
    "          \"trueNegativeCount\": \"484\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.88389677,\n",
    "          \"recall\": 0.83977497,\n",
    "          \"precision\": 0.98338586,\n",
    "          \"f1Score\": 0.90592426,\n",
    "          \"falsePositiveRate\": 0.10526316,\n",
    "          \"truePositiveCount\": \"3433\",\n",
    "          \"falsePositiveCount\": \"58\",\n",
    "          \"falseNegativeCount\": \"655\",\n",
    "          \"trueNegativeCount\": \"493\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.89513993,\n",
    "          \"recall\": 0.8299902,\n",
    "          \"precision\": 0.9840487,\n",
    "          \"f1Score\": 0.9004777,\n",
    "          \"falsePositiveRate\": 0.09981851,\n",
    "          \"truePositiveCount\": \"3393\",\n",
    "          \"falsePositiveCount\": \"55\",\n",
    "          \"falseNegativeCount\": \"695\",\n",
    "          \"trueNegativeCount\": \"496\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.90273213,\n",
    "          \"recall\": 0.82020545,\n",
    "          \"precision\": 0.9853071,\n",
    "          \"f1Score\": 0.8952076,\n",
    "          \"falsePositiveRate\": 0.0907441,\n",
    "          \"truePositiveCount\": \"3353\",\n",
    "          \"falsePositiveCount\": \"50\",\n",
    "          \"falseNegativeCount\": \"735\",\n",
    "          \"trueNegativeCount\": \"501\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9118437,\n",
    "          \"recall\": 0.81017613,\n",
    "          \"precision\": 0.98718333,\n",
    "          \"f1Score\": 0.88996375,\n",
    "          \"falsePositiveRate\": 0.07803993,\n",
    "          \"truePositiveCount\": \"3312\",\n",
    "          \"falsePositiveCount\": \"43\",\n",
    "          \"falseNegativeCount\": \"776\",\n",
    "          \"trueNegativeCount\": \"508\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9204901,\n",
    "          \"recall\": 0.79990214,\n",
    "          \"precision\": 0.98821396,\n",
    "          \"f1Score\": 0.8841422,\n",
    "          \"falsePositiveRate\": 0.0707804,\n",
    "          \"truePositiveCount\": \"3270\",\n",
    "          \"falsePositiveCount\": \"39\",\n",
    "          \"falseNegativeCount\": \"818\",\n",
    "          \"trueNegativeCount\": \"512\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9283992,\n",
    "          \"recall\": 0.7898728,\n",
    "          \"precision\": 0.9898835,\n",
    "          \"f1Score\": 0.87863946,\n",
    "          \"falsePositiveRate\": 0.05989111,\n",
    "          \"truePositiveCount\": \"3229\",\n",
    "          \"falsePositiveCount\": \"33\",\n",
    "          \"falseNegativeCount\": \"859\",\n",
    "          \"trueNegativeCount\": \"518\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.934707,\n",
    "          \"recall\": 0.77984345,\n",
    "          \"precision\": 0.9909854,\n",
    "          \"f1Score\": 0.8728268,\n",
    "          \"falsePositiveRate\": 0.05263158,\n",
    "          \"truePositiveCount\": \"3188\",\n",
    "          \"falsePositiveCount\": \"29\",\n",
    "          \"falseNegativeCount\": \"900\",\n",
    "          \"trueNegativeCount\": \"522\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9401358,\n",
    "          \"recall\": 0.7698141,\n",
    "          \"precision\": 0.99211854,\n",
    "          \"f1Score\": 0.86694217,\n",
    "          \"falsePositiveRate\": 0.04537205,\n",
    "          \"truePositiveCount\": \"3147\",\n",
    "          \"falsePositiveCount\": \"25\",\n",
    "          \"falseNegativeCount\": \"941\",\n",
    "          \"trueNegativeCount\": \"526\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.943966,\n",
    "          \"recall\": 0.7600294,\n",
    "          \"precision\": 0.9923347,\n",
    "          \"f1Score\": 0.86078405,\n",
    "          \"falsePositiveRate\": 0.043557167,\n",
    "          \"truePositiveCount\": \"3107\",\n",
    "          \"falsePositiveCount\": \"24\",\n",
    "          \"falseNegativeCount\": \"981\",\n",
    "          \"trueNegativeCount\": \"527\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9484665,\n",
    "          \"recall\": 0.7497554,\n",
    "          \"precision\": 0.9925518,\n",
    "          \"f1Score\": 0.85423636,\n",
    "          \"falsePositiveRate\": 0.041742288,\n",
    "          \"truePositiveCount\": \"3065\",\n",
    "          \"falsePositiveCount\": \"23\",\n",
    "          \"falseNegativeCount\": \"1023\",\n",
    "          \"trueNegativeCount\": \"528\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9512129,\n",
    "          \"recall\": 0.7399706,\n",
    "          \"precision\": 0.9931057,\n",
    "          \"f1Score\": 0.8480516,\n",
    "          \"falsePositiveRate\": 0.03811252,\n",
    "          \"truePositiveCount\": \"3025\",\n",
    "          \"falsePositiveCount\": \"21\",\n",
    "          \"falseNegativeCount\": \"1063\",\n",
    "          \"trueNegativeCount\": \"530\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95533496,\n",
    "          \"recall\": 0.7299413,\n",
    "          \"precision\": 0.9946667,\n",
    "          \"f1Score\": 0.8419865,\n",
    "          \"falsePositiveRate\": 0.029038113,\n",
    "          \"truePositiveCount\": \"2984\",\n",
    "          \"falsePositiveCount\": \"16\",\n",
    "          \"falseNegativeCount\": \"1104\",\n",
    "          \"trueNegativeCount\": \"535\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95724624,\n",
    "          \"recall\": 0.72015655,\n",
    "          \"precision\": 0.9956037,\n",
    "          \"f1Score\": 0.8357701,\n",
    "          \"falsePositiveRate\": 0.023593467,\n",
    "          \"truePositiveCount\": \"2944\",\n",
    "          \"falsePositiveCount\": \"13\",\n",
    "          \"falseNegativeCount\": \"1144\",\n",
    "          \"trueNegativeCount\": \"538\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96008635,\n",
    "          \"recall\": 0.7101272,\n",
    "          \"precision\": 0.99588335,\n",
    "          \"f1Score\": 0.82907325,\n",
    "          \"falsePositiveRate\": 0.021778584,\n",
    "          \"truePositiveCount\": \"2903\",\n",
    "          \"falsePositiveCount\": \"12\",\n",
    "          \"falseNegativeCount\": \"1185\",\n",
    "          \"trueNegativeCount\": \"539\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9629221,\n",
    "          \"recall\": 0.70009786,\n",
    "          \"precision\": 0.99617124,\n",
    "          \"f1Score\": 0.82229567,\n",
    "          \"falsePositiveRate\": 0.019963702,\n",
    "          \"truePositiveCount\": \"2862\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1226\",\n",
    "          \"trueNegativeCount\": \"540\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9646139,\n",
    "          \"recall\": 0.6900685,\n",
    "          \"precision\": 0.9961158,\n",
    "          \"f1Score\": 0.8153179,\n",
    "          \"falsePositiveRate\": 0.019963702,\n",
    "          \"truePositiveCount\": \"2821\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1267\",\n",
    "          \"trueNegativeCount\": \"540\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96656674,\n",
    "          \"recall\": 0.67979455,\n",
    "          \"precision\": 0.99605733,\n",
    "          \"f1Score\": 0.8080838,\n",
    "          \"falsePositiveRate\": 0.019963702,\n",
    "          \"truePositiveCount\": \"2779\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1309\",\n",
    "          \"trueNegativeCount\": \"540\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96897614,\n",
    "          \"recall\": 0.6700098,\n",
    "          \"precision\": 0.9963623,\n",
    "          \"f1Score\": 0.8012286,\n",
    "          \"falsePositiveRate\": 0.01814882,\n",
    "          \"truePositiveCount\": \"2739\",\n",
    "          \"falsePositiveCount\": \"10\",\n",
    "          \"falseNegativeCount\": \"1349\",\n",
    "          \"trueNegativeCount\": \"541\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9706901,\n",
    "          \"recall\": 0.6599804,\n",
    "          \"precision\": 0.99630725,\n",
    "          \"f1Score\": 0.79399645,\n",
    "          \"falsePositiveRate\": 0.01814882,\n",
    "          \"truePositiveCount\": \"2698\",\n",
    "          \"falsePositiveCount\": \"10\",\n",
    "          \"falseNegativeCount\": \"1390\",\n",
    "          \"trueNegativeCount\": \"541\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97229564,\n",
    "          \"recall\": 0.6499511,\n",
    "          \"precision\": 0.99699813,\n",
    "          \"f1Score\": 0.7869095,\n",
    "          \"falsePositiveRate\": 0.014519056,\n",
    "          \"truePositiveCount\": \"2657\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"1431\",\n",
    "          \"trueNegativeCount\": \"543\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9738624,\n",
    "          \"recall\": 0.6399217,\n",
    "          \"precision\": 0.9969512,\n",
    "          \"f1Score\": 0.7794994,\n",
    "          \"falsePositiveRate\": 0.014519056,\n",
    "          \"truePositiveCount\": \"2616\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"1472\",\n",
    "          \"trueNegativeCount\": \"543\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97582275,\n",
    "          \"recall\": 0.62989235,\n",
    "          \"precision\": 0.99728894,\n",
    "          \"f1Score\": 0.7721139,\n",
    "          \"falsePositiveRate\": 0.012704174,\n",
    "          \"truePositiveCount\": \"2575\",\n",
    "          \"falsePositiveCount\": \"7\",\n",
    "          \"falseNegativeCount\": \"1513\",\n",
    "          \"trueNegativeCount\": \"544\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9772092,\n",
    "          \"recall\": 0.62010765,\n",
    "          \"precision\": 0.9976387,\n",
    "          \"f1Score\": 0.76482123,\n",
    "          \"falsePositiveRate\": 0.010889292,\n",
    "          \"truePositiveCount\": \"2535\",\n",
    "          \"falsePositiveCount\": \"6\",\n",
    "          \"falseNegativeCount\": \"1553\",\n",
    "          \"trueNegativeCount\": \"545\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9781838,\n",
    "          \"recall\": 0.6100783,\n",
    "          \"precision\": 0.9979992,\n",
    "          \"f1Score\": 0.7572491,\n",
    "          \"falsePositiveRate\": 0.00907441,\n",
    "          \"truePositiveCount\": \"2494\",\n",
    "          \"falsePositiveCount\": \"5\",\n",
    "          \"falseNegativeCount\": \"1594\",\n",
    "          \"trueNegativeCount\": \"546\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9794321,\n",
    "          \"recall\": 0.5998043,\n",
    "          \"precision\": 0.99837136,\n",
    "          \"f1Score\": 0.74938875,\n",
    "          \"falsePositiveRate\": 0.007259528,\n",
    "          \"truePositiveCount\": \"2452\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"1636\",\n",
    "          \"trueNegativeCount\": \"547\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9804444,\n",
    "          \"recall\": 0.5900196,\n",
    "          \"precision\": 0.99834436,\n",
    "          \"f1Score\": 0.74169743,\n",
    "          \"falsePositiveRate\": 0.007259528,\n",
    "          \"truePositiveCount\": \"2412\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"1676\",\n",
    "          \"trueNegativeCount\": \"547\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9813577,\n",
    "          \"recall\": 0.5799902,\n",
    "          \"precision\": 0.9991572,\n",
    "          \"f1Score\": 0.7339421,\n",
    "          \"falsePositiveRate\": 0.003629764,\n",
    "          \"truePositiveCount\": \"2371\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"1717\",\n",
    "          \"trueNegativeCount\": \"549\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9822546,\n",
    "          \"recall\": 0.56996083,\n",
    "          \"precision\": 0.99914235,\n",
    "          \"f1Score\": 0.7258567,\n",
    "          \"falsePositiveRate\": 0.003629764,\n",
    "          \"truePositiveCount\": \"2330\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"1758\",\n",
    "          \"trueNegativeCount\": \"549\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9832565,\n",
    "          \"recall\": 0.5599315,\n",
    "          \"precision\": 0.99912703,\n",
    "          \"f1Score\": 0.71766734,\n",
    "          \"falsePositiveRate\": 0.003629764,\n",
    "          \"truePositiveCount\": \"2289\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"1799\",\n",
    "          \"trueNegativeCount\": \"549\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.983971,\n",
    "          \"recall\": 0.54990214,\n",
    "          \"precision\": 0.99955535,\n",
    "          \"f1Score\": 0.709484,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2248\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1840\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98478603,\n",
    "          \"recall\": 0.54011744,\n",
    "          \"precision\": 0.9995473,\n",
    "          \"f1Score\": 0.7012863,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2208\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1880\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98576885,\n",
    "          \"recall\": 0.52984345,\n",
    "          \"precision\": 0.99953854,\n",
    "          \"f1Score\": 0.6925659,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2166\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1922\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9865271,\n",
    "          \"recall\": 0.5198141,\n",
    "          \"precision\": 0.99952966,\n",
    "          \"f1Score\": 0.6839395,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2125\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1963\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98722655,\n",
    "          \"recall\": 0.5100294,\n",
    "          \"precision\": 0.9995206,\n",
    "          \"f1Score\": 0.675413,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2085\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2003\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98776835,\n",
    "          \"recall\": 0.49975538,\n",
    "          \"precision\": 0.99951077,\n",
    "          \"f1Score\": 0.66634053,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2043\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2045\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98843795,\n",
    "          \"recall\": 0.49021527,\n",
    "          \"precision\": 0.9995012,\n",
    "          \"f1Score\": 0.657804,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2004\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2084\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98898214,\n",
    "          \"recall\": 0.4801859,\n",
    "          \"precision\": 0.99949086,\n",
    "          \"f1Score\": 0.64871114,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1963\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2125\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98952276,\n",
    "          \"recall\": 0.46991193,\n",
    "          \"precision\": 0.9994797,\n",
    "          \"f1Score\": 0.63926786,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1921\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2167\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99007314,\n",
    "          \"recall\": 0.4598826,\n",
    "          \"precision\": 0.9994684,\n",
    "          \"f1Score\": 0.62992126,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1880\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2208\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.990457,\n",
    "          \"recall\": 0.45009786,\n",
    "          \"precision\": 0.9994568,\n",
    "          \"f1Score\": 0.620678,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1840\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2248\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99079466,\n",
    "          \"recall\": 0.44006848,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.61117715,\n",
    "          \"truePositiveCount\": \"1799\",\n",
    "          \"falseNegativeCount\": \"2289\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9911339,\n",
    "          \"recall\": 0.43003914,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.60143685,\n",
    "          \"truePositiveCount\": \"1758\",\n",
    "          \"falseNegativeCount\": \"2330\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99141955,\n",
    "          \"recall\": 0.4200098,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.591559,\n",
    "          \"truePositiveCount\": \"1717\",\n",
    "          \"falseNegativeCount\": \"2371\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9917892,\n",
    "          \"recall\": 0.40998042,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5815406,\n",
    "          \"truePositiveCount\": \"1676\",\n",
    "          \"falseNegativeCount\": \"2412\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9921363,\n",
    "          \"recall\": 0.39995107,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.57137865,\n",
    "          \"truePositiveCount\": \"1635\",\n",
    "          \"falseNegativeCount\": \"2453\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99253094,\n",
    "          \"recall\": 0.38992172,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.56107,\n",
    "          \"truePositiveCount\": \"1594\",\n",
    "          \"falseNegativeCount\": \"2494\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9928577,\n",
    "          \"recall\": 0.380137,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5508685,\n",
    "          \"truePositiveCount\": \"1554\",\n",
    "          \"falseNegativeCount\": \"2534\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99313205,\n",
    "          \"recall\": 0.369863,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.54,\n",
    "          \"truePositiveCount\": \"1512\",\n",
    "          \"falseNegativeCount\": \"2576\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9933454,\n",
    "          \"recall\": 0.35983366,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5292319,\n",
    "          \"truePositiveCount\": \"1471\",\n",
    "          \"falseNegativeCount\": \"2617\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99355274,\n",
    "          \"recall\": 0.35004893,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5185722,\n",
    "          \"truePositiveCount\": \"1431\",\n",
    "          \"falseNegativeCount\": \"2657\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99378794,\n",
    "          \"recall\": 0.33977494,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.507212,\n",
    "          \"truePositiveCount\": \"1389\",\n",
    "          \"falseNegativeCount\": \"2699\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9939992,\n",
    "          \"recall\": 0.33023483,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.49650607,\n",
    "          \"truePositiveCount\": \"1350\",\n",
    "          \"falseNegativeCount\": \"2738\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9942213,\n",
    "          \"recall\": 0.31996086,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.48480356,\n",
    "          \"truePositiveCount\": \"1308\",\n",
    "          \"falseNegativeCount\": \"2780\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9944552,\n",
    "          \"recall\": 0.30993152,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.47320262,\n",
    "          \"truePositiveCount\": \"1267\",\n",
    "          \"falseNegativeCount\": \"2821\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9946339,\n",
    "          \"recall\": 0.29990214,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.46142265,\n",
    "          \"truePositiveCount\": \"1226\",\n",
    "          \"falseNegativeCount\": \"2862\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99486357,\n",
    "          \"recall\": 0.2898728,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.44945952,\n",
    "          \"truePositiveCount\": \"1185\",\n",
    "          \"falseNegativeCount\": \"2903\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9950511,\n",
    "          \"recall\": 0.28008807,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.4376075,\n",
    "          \"truePositiveCount\": \"1145\",\n",
    "          \"falseNegativeCount\": \"2943\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9953063,\n",
    "          \"recall\": 0.2698141,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.42496628,\n",
    "          \"truePositiveCount\": \"1103\",\n",
    "          \"falseNegativeCount\": \"2985\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99548787,\n",
    "          \"recall\": 0.26002935,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.4127354,\n",
    "          \"truePositiveCount\": \"1063\",\n",
    "          \"falseNegativeCount\": \"3025\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99564606,\n",
    "          \"recall\": 0.25024462,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.40031305,\n",
    "          \"truePositiveCount\": \"1023\",\n",
    "          \"falseNegativeCount\": \"3065\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9958494,\n",
    "          \"recall\": 0.23997064,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.3870586,\n",
    "          \"truePositiveCount\": \"981\",\n",
    "          \"falseNegativeCount\": \"3107\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9959897,\n",
    "          \"recall\": 0.2299413,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.37390614,\n",
    "          \"truePositiveCount\": \"940\",\n",
    "          \"falseNegativeCount\": \"3148\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9961461,\n",
    "          \"recall\": 0.21991193,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.3605374,\n",
    "          \"truePositiveCount\": \"899\",\n",
    "          \"falseNegativeCount\": \"3189\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99626255,\n",
    "          \"recall\": 0.20988259,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.346947,\n",
    "          \"truePositiveCount\": \"858\",\n",
    "          \"falseNegativeCount\": \"3230\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99635655,\n",
    "          \"recall\": 0.19985323,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.33312947,\n",
    "          \"truePositiveCount\": \"817\",\n",
    "          \"falseNegativeCount\": \"3271\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9964815,\n",
    "          \"recall\": 0.1900685,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.31942445,\n",
    "          \"truePositiveCount\": \"777\",\n",
    "          \"falseNegativeCount\": \"3311\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9966108,\n",
    "          \"recall\": 0.17979452,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.30478954,\n",
    "          \"truePositiveCount\": \"735\",\n",
    "          \"falseNegativeCount\": \"3353\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99673027,\n",
    "          \"recall\": 0.16976516,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.29025513,\n",
    "          \"truePositiveCount\": \"694\",\n",
    "          \"falseNegativeCount\": \"3394\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99681914,\n",
    "          \"recall\": 0.15998043,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.27583298,\n",
    "          \"truePositiveCount\": \"654\",\n",
    "          \"falseNegativeCount\": \"3434\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9969455,\n",
    "          \"recall\": 0.15019569,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.26116547,\n",
    "          \"truePositiveCount\": \"614\",\n",
    "          \"falseNegativeCount\": \"3474\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99705213,\n",
    "          \"recall\": 0.13992172,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.24549356,\n",
    "          \"truePositiveCount\": \"572\",\n",
    "          \"falseNegativeCount\": \"3516\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9971576,\n",
    "          \"recall\": 0.13013698,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.23030303,\n",
    "          \"truePositiveCount\": \"532\",\n",
    "          \"falseNegativeCount\": \"3556\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9972461,\n",
    "          \"recall\": 0.11986301,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.21406728,\n",
    "          \"truePositiveCount\": \"490\",\n",
    "          \"falseNegativeCount\": \"3598\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9973425,\n",
    "          \"recall\": 0.110078275,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.19832525,\n",
    "          \"truePositiveCount\": \"450\",\n",
    "          \"falseNegativeCount\": \"3638\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9974355,\n",
    "          \"recall\": 0.10004892,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.18189904,\n",
    "          \"truePositiveCount\": \"409\",\n",
    "          \"falseNegativeCount\": \"3679\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9975045,\n",
    "          \"recall\": 0.09001957,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.16517055,\n",
    "          \"truePositiveCount\": \"368\",\n",
    "          \"falseNegativeCount\": \"3720\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99760437,\n",
    "          \"recall\": 0.079990216,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.14813137,\n",
    "          \"truePositiveCount\": \"327\",\n",
    "          \"falseNegativeCount\": \"3761\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9976864,\n",
    "          \"recall\": 0.07020548,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.1312,\n",
    "          \"truePositiveCount\": \"287\",\n",
    "          \"falseNegativeCount\": \"3801\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9977548,\n",
    "          \"recall\": 0.059931505,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.11308562,\n",
    "          \"truePositiveCount\": \"245\",\n",
    "          \"falseNegativeCount\": \"3843\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99784184,\n",
    "          \"recall\": 0.05014677,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.095504306,\n",
    "          \"truePositiveCount\": \"205\",\n",
    "          \"falseNegativeCount\": \"3883\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9979226,\n",
    "          \"recall\": 0.040117417,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.07714017,\n",
    "          \"truePositiveCount\": \"164\",\n",
    "          \"falseNegativeCount\": \"3924\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9980229,\n",
    "          \"recall\": 0.030088063,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.058418427,\n",
    "          \"truePositiveCount\": \"123\",\n",
    "          \"falseNegativeCount\": \"3965\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9981275,\n",
    "          \"recall\": 0.01981409,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.03885824,\n",
    "          \"truePositiveCount\": \"81\",\n",
    "          \"falseNegativeCount\": \"4007\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9982374,\n",
    "          \"recall\": 0.009784736,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.019379845,\n",
    "          \"truePositiveCount\": \"40\",\n",
    "          \"falseNegativeCount\": \"4048\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9987167,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4088\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 1.0,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4088\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        }\n",
    "      ],\n",
    "      \"auRoc\": 0.9340445,\n",
    "      \"logLoss\": 0.09388557\n",
    "    },\n",
    "    \"displayName\": \"1\"\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/1660852072267559125\",\n",
    "    \"annotationSpecId\": \"not available\",\n",
    "    \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "    \"evaluatedExampleCount\": 551,\n",
    "    \"classificationEvaluationMetrics\": {\n",
    "      \"auPrc\": 0.61527693,\n",
    "      \"confidenceMetricsEntry\": [\n",
    "        {\n",
    "          \"recall\": 1.0,\n",
    "          \"precision\": 0.1187756,\n",
    "          \"f1Score\": 0.2123314,\n",
    "          \"falsePositiveRate\": 1.0,\n",
    "          \"truePositiveCount\": \"551\",\n",
    "          \"falsePositiveCount\": \"4088\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.009267787,\n",
    "          \"recall\": 0.9981851,\n",
    "          \"precision\": 0.19400352,\n",
    "          \"f1Score\": 0.3248671,\n",
    "          \"falsePositiveRate\": 0.55895305,\n",
    "          \"truePositiveCount\": \"550\",\n",
    "          \"falsePositiveCount\": \"2285\",\n",
    "          \"falseNegativeCount\": \"1\",\n",
    "          \"trueNegativeCount\": \"1803\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.022539802,\n",
    "          \"recall\": 0.9891107,\n",
    "          \"precision\": 0.2587844,\n",
    "          \"f1Score\": 0.4102371,\n",
    "          \"falsePositiveRate\": 0.38184932,\n",
    "          \"truePositiveCount\": \"545\",\n",
    "          \"falsePositiveCount\": \"1561\",\n",
    "          \"falseNegativeCount\": \"6\",\n",
    "          \"trueNegativeCount\": \"2527\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.03954769,\n",
    "          \"recall\": 0.9782214,\n",
    "          \"precision\": 0.3113807,\n",
    "          \"f1Score\": 0.47239265,\n",
    "          \"falsePositiveRate\": 0.29158512,\n",
    "          \"truePositiveCount\": \"539\",\n",
    "          \"falsePositiveCount\": \"1192\",\n",
    "          \"falseNegativeCount\": \"12\",\n",
    "          \"trueNegativeCount\": \"2896\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.045538254,\n",
    "          \"recall\": 0.969147,\n",
    "          \"precision\": 0.32902032,\n",
    "          \"f1Score\": 0.49126035,\n",
    "          \"falsePositiveRate\": 0.26638943,\n",
    "          \"truePositiveCount\": \"534\",\n",
    "          \"falsePositiveCount\": \"1089\",\n",
    "          \"falseNegativeCount\": \"17\",\n",
    "          \"trueNegativeCount\": \"2999\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.050852664,\n",
    "          \"recall\": 0.95825773,\n",
    "          \"precision\": 0.33802816,\n",
    "          \"f1Score\": 0.49976337,\n",
    "          \"falsePositiveRate\": 0.2529354,\n",
    "          \"truePositiveCount\": \"528\",\n",
    "          \"falsePositiveCount\": \"1034\",\n",
    "          \"falseNegativeCount\": \"23\",\n",
    "          \"trueNegativeCount\": \"3054\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.063637815,\n",
    "          \"recall\": 0.9491833,\n",
    "          \"precision\": 0.3647141,\n",
    "          \"f1Score\": 0.52695215,\n",
    "          \"falsePositiveRate\": 0.22284736,\n",
    "          \"truePositiveCount\": \"523\",\n",
    "          \"falsePositiveCount\": \"911\",\n",
    "          \"falseNegativeCount\": \"28\",\n",
    "          \"trueNegativeCount\": \"3177\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.07210142,\n",
    "          \"recall\": 0.938294,\n",
    "          \"precision\": 0.377097,\n",
    "          \"f1Score\": 0.5379813,\n",
    "          \"falsePositiveRate\": 0.2089041,\n",
    "          \"truePositiveCount\": \"517\",\n",
    "          \"falsePositiveCount\": \"854\",\n",
    "          \"falseNegativeCount\": \"34\",\n",
    "          \"trueNegativeCount\": \"3234\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.07750216,\n",
    "          \"recall\": 0.9292196,\n",
    "          \"precision\": 0.3826607,\n",
    "          \"f1Score\": 0.54208577,\n",
    "          \"falsePositiveRate\": 0.2020548,\n",
    "          \"truePositiveCount\": \"512\",\n",
    "          \"falsePositiveCount\": \"826\",\n",
    "          \"falseNegativeCount\": \"39\",\n",
    "          \"trueNegativeCount\": \"3262\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.09185968,\n",
    "          \"recall\": 0.9183303,\n",
    "          \"precision\": 0.4,\n",
    "          \"f1Score\": 0.55726874,\n",
    "          \"falsePositiveRate\": 0.18566537,\n",
    "          \"truePositiveCount\": \"506\",\n",
    "          \"falsePositiveCount\": \"759\",\n",
    "          \"falseNegativeCount\": \"45\",\n",
    "          \"trueNegativeCount\": \"3329\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.09674809,\n",
    "          \"recall\": 0.9092559,\n",
    "          \"precision\": 0.40501213,\n",
    "          \"f1Score\": 0.5604027,\n",
    "          \"falsePositiveRate\": 0.18003914,\n",
    "          \"truePositiveCount\": \"501\",\n",
    "          \"falsePositiveCount\": \"736\",\n",
    "          \"falseNegativeCount\": \"50\",\n",
    "          \"trueNegativeCount\": \"3352\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.10816234,\n",
    "          \"recall\": 0.89836663,\n",
    "          \"precision\": 0.4212766,\n",
    "          \"f1Score\": 0.5735805,\n",
    "          \"falsePositiveRate\": 0.16634052,\n",
    "          \"truePositiveCount\": \"495\",\n",
    "          \"falsePositiveCount\": \"680\",\n",
    "          \"falseNegativeCount\": \"56\",\n",
    "          \"trueNegativeCount\": \"3408\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.12236822,\n",
    "          \"recall\": 0.8892922,\n",
    "          \"precision\": 0.43672013,\n",
    "          \"f1Score\": 0.58577406,\n",
    "          \"falsePositiveRate\": 0.15459883,\n",
    "          \"truePositiveCount\": \"490\",\n",
    "          \"falsePositiveCount\": \"632\",\n",
    "          \"falseNegativeCount\": \"61\",\n",
    "          \"trueNegativeCount\": \"3456\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.1267207,\n",
    "          \"recall\": 0.8784029,\n",
    "          \"precision\": 0.4368231,\n",
    "          \"f1Score\": 0.58348405,\n",
    "          \"falsePositiveRate\": 0.15264188,\n",
    "          \"truePositiveCount\": \"484\",\n",
    "          \"falsePositiveCount\": \"624\",\n",
    "          \"falseNegativeCount\": \"67\",\n",
    "          \"trueNegativeCount\": \"3464\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.15273023,\n",
    "          \"recall\": 0.8693285,\n",
    "          \"precision\": 0.46550047,\n",
    "          \"f1Score\": 0.60632914,\n",
    "          \"falsePositiveRate\": 0.13454011,\n",
    "          \"truePositiveCount\": \"479\",\n",
    "          \"falsePositiveCount\": \"550\",\n",
    "          \"falseNegativeCount\": \"72\",\n",
    "          \"trueNegativeCount\": \"3538\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.1649416,\n",
    "          \"recall\": 0.8584392,\n",
    "          \"precision\": 0.47252747,\n",
    "          \"f1Score\": 0.6095361,\n",
    "          \"falsePositiveRate\": 0.12915851,\n",
    "          \"truePositiveCount\": \"473\",\n",
    "          \"falsePositiveCount\": \"528\",\n",
    "          \"falseNegativeCount\": \"78\",\n",
    "          \"trueNegativeCount\": \"3560\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.18867286,\n",
    "          \"recall\": 0.8493648,\n",
    "          \"precision\": 0.49628845,\n",
    "          \"f1Score\": 0.62650603,\n",
    "          \"falsePositiveRate\": 0.116193734,\n",
    "          \"truePositiveCount\": \"468\",\n",
    "          \"falsePositiveCount\": \"475\",\n",
    "          \"falseNegativeCount\": \"83\",\n",
    "          \"trueNegativeCount\": \"3613\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.19824302,\n",
    "          \"recall\": 0.8384755,\n",
    "          \"precision\": 0.5038168,\n",
    "          \"f1Score\": 0.6294278,\n",
    "          \"falsePositiveRate\": 0.11130137,\n",
    "          \"truePositiveCount\": \"462\",\n",
    "          \"falsePositiveCount\": \"455\",\n",
    "          \"falseNegativeCount\": \"89\",\n",
    "          \"trueNegativeCount\": \"3633\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.20855853,\n",
    "          \"recall\": 0.8294011,\n",
    "          \"precision\": 0.5163842,\n",
    "          \"f1Score\": 0.6364902,\n",
    "          \"falsePositiveRate\": 0.104696676,\n",
    "          \"truePositiveCount\": \"457\",\n",
    "          \"falsePositiveCount\": \"428\",\n",
    "          \"falseNegativeCount\": \"94\",\n",
    "          \"trueNegativeCount\": \"3660\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2164171,\n",
    "          \"recall\": 0.8185118,\n",
    "          \"precision\": 0.51898736,\n",
    "          \"f1Score\": 0.6352113,\n",
    "          \"falsePositiveRate\": 0.10225049,\n",
    "          \"truePositiveCount\": \"451\",\n",
    "          \"falsePositiveCount\": \"418\",\n",
    "          \"falseNegativeCount\": \"100\",\n",
    "          \"trueNegativeCount\": \"3670\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2228882,\n",
    "          \"recall\": 0.8094374,\n",
    "          \"precision\": 0.52347416,\n",
    "          \"f1Score\": 0.63578045,\n",
    "          \"falsePositiveRate\": 0.09931507,\n",
    "          \"truePositiveCount\": \"446\",\n",
    "          \"falsePositiveCount\": \"406\",\n",
    "          \"falseNegativeCount\": \"105\",\n",
    "          \"trueNegativeCount\": \"3682\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2343938,\n",
    "          \"recall\": 0.7985481,\n",
    "          \"precision\": 0.5339806,\n",
    "          \"f1Score\": 0.64,\n",
    "          \"falsePositiveRate\": 0.09393346,\n",
    "          \"truePositiveCount\": \"440\",\n",
    "          \"falsePositiveCount\": \"384\",\n",
    "          \"falseNegativeCount\": \"111\",\n",
    "          \"trueNegativeCount\": \"3704\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.24437484,\n",
    "          \"recall\": 0.7894737,\n",
    "          \"precision\": 0.5390335,\n",
    "          \"f1Score\": 0.640648,\n",
    "          \"falsePositiveRate\": 0.090998046,\n",
    "          \"truePositiveCount\": \"435\",\n",
    "          \"falsePositiveCount\": \"372\",\n",
    "          \"falseNegativeCount\": \"116\",\n",
    "          \"trueNegativeCount\": \"3716\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.24899487,\n",
    "          \"recall\": 0.7785844,\n",
    "          \"precision\": 0.53692114,\n",
    "          \"f1Score\": 0.63555557,\n",
    "          \"falsePositiveRate\": 0.0905088,\n",
    "          \"truePositiveCount\": \"429\",\n",
    "          \"falsePositiveCount\": \"370\",\n",
    "          \"falseNegativeCount\": \"122\",\n",
    "          \"trueNegativeCount\": \"3718\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.25527748,\n",
    "          \"recall\": 0.76951,\n",
    "          \"precision\": 0.5401274,\n",
    "          \"f1Score\": 0.6347305,\n",
    "          \"falsePositiveRate\": 0.08830724,\n",
    "          \"truePositiveCount\": \"424\",\n",
    "          \"falsePositiveCount\": \"361\",\n",
    "          \"falseNegativeCount\": \"127\",\n",
    "          \"trueNegativeCount\": \"3727\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2643137,\n",
    "          \"recall\": 0.7586207,\n",
    "          \"precision\": 0.5414508,\n",
    "          \"f1Score\": 0.6318972,\n",
    "          \"falsePositiveRate\": 0.08659491,\n",
    "          \"truePositiveCount\": \"418\",\n",
    "          \"falsePositiveCount\": \"354\",\n",
    "          \"falseNegativeCount\": \"133\",\n",
    "          \"trueNegativeCount\": \"3734\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.27015588,\n",
    "          \"recall\": 0.7495463,\n",
    "          \"precision\": 0.544137,\n",
    "          \"f1Score\": 0.63053435,\n",
    "          \"falsePositiveRate\": 0.08463796,\n",
    "          \"truePositiveCount\": \"413\",\n",
    "          \"falsePositiveCount\": \"346\",\n",
    "          \"falseNegativeCount\": \"138\",\n",
    "          \"trueNegativeCount\": \"3742\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2785622,\n",
    "          \"recall\": 0.738657,\n",
    "          \"precision\": 0.547043,\n",
    "          \"f1Score\": 0.62857145,\n",
    "          \"falsePositiveRate\": 0.0824364,\n",
    "          \"truePositiveCount\": \"407\",\n",
    "          \"falsePositiveCount\": \"337\",\n",
    "          \"falseNegativeCount\": \"144\",\n",
    "          \"trueNegativeCount\": \"3751\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.29147997,\n",
    "          \"recall\": 0.72958255,\n",
    "          \"precision\": 0.5552486,\n",
    "          \"f1Score\": 0.63058823,\n",
    "          \"falsePositiveRate\": 0.07876712,\n",
    "          \"truePositiveCount\": \"402\",\n",
    "          \"falsePositiveCount\": \"322\",\n",
    "          \"falseNegativeCount\": \"149\",\n",
    "          \"trueNegativeCount\": \"3766\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.30125153,\n",
    "          \"recall\": 0.71869326,\n",
    "          \"precision\": 0.55774647,\n",
    "          \"f1Score\": 0.628073,\n",
    "          \"falsePositiveRate\": 0.07681017,\n",
    "          \"truePositiveCount\": \"396\",\n",
    "          \"falsePositiveCount\": \"314\",\n",
    "          \"falseNegativeCount\": \"155\",\n",
    "          \"trueNegativeCount\": \"3774\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.30796623,\n",
    "          \"recall\": 0.70961887,\n",
    "          \"precision\": 0.55857146,\n",
    "          \"f1Score\": 0.6250999,\n",
    "          \"falsePositiveRate\": 0.07558709,\n",
    "          \"truePositiveCount\": \"391\",\n",
    "          \"falsePositiveCount\": \"309\",\n",
    "          \"falseNegativeCount\": \"160\",\n",
    "          \"trueNegativeCount\": \"3779\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.32498023,\n",
    "          \"recall\": 0.6987296,\n",
    "          \"precision\": 0.5636896,\n",
    "          \"f1Score\": 0.623987,\n",
    "          \"falsePositiveRate\": 0.07289628,\n",
    "          \"truePositiveCount\": \"385\",\n",
    "          \"falsePositiveCount\": \"298\",\n",
    "          \"falseNegativeCount\": \"166\",\n",
    "          \"trueNegativeCount\": \"3790\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.33223236,\n",
    "          \"recall\": 0.6896552,\n",
    "          \"precision\": 0.5654762,\n",
    "          \"f1Score\": 0.6214227,\n",
    "          \"falsePositiveRate\": 0.071428575,\n",
    "          \"truePositiveCount\": \"380\",\n",
    "          \"falsePositiveCount\": \"292\",\n",
    "          \"falseNegativeCount\": \"171\",\n",
    "          \"trueNegativeCount\": \"3796\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.3348615,\n",
    "          \"recall\": 0.6787659,\n",
    "          \"precision\": 0.562406,\n",
    "          \"f1Score\": 0.61513156,\n",
    "          \"falsePositiveRate\": 0.07118395,\n",
    "          \"truePositiveCount\": \"374\",\n",
    "          \"falsePositiveCount\": \"291\",\n",
    "          \"falseNegativeCount\": \"177\",\n",
    "          \"trueNegativeCount\": \"3797\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.33746937,\n",
    "          \"recall\": 0.66969144,\n",
    "          \"precision\": 0.5607903,\n",
    "          \"f1Score\": 0.61042184,\n",
    "          \"falsePositiveRate\": 0.070694715,\n",
    "          \"truePositiveCount\": \"369\",\n",
    "          \"falsePositiveCount\": \"289\",\n",
    "          \"falseNegativeCount\": \"182\",\n",
    "          \"trueNegativeCount\": \"3799\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.34359238,\n",
    "          \"recall\": 0.65880215,\n",
    "          \"precision\": 0.561051,\n",
    "          \"f1Score\": 0.60601,\n",
    "          \"falsePositiveRate\": 0.06947163,\n",
    "          \"truePositiveCount\": \"363\",\n",
    "          \"falsePositiveCount\": \"284\",\n",
    "          \"falseNegativeCount\": \"188\",\n",
    "          \"trueNegativeCount\": \"3804\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.3495217,\n",
    "          \"recall\": 0.64972776,\n",
    "          \"precision\": 0.5585023,\n",
    "          \"f1Score\": 0.6006711,\n",
    "          \"falsePositiveRate\": 0.069227,\n",
    "          \"truePositiveCount\": \"358\",\n",
    "          \"falsePositiveCount\": \"283\",\n",
    "          \"falseNegativeCount\": \"193\",\n",
    "          \"trueNegativeCount\": \"3805\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.35605764,\n",
    "          \"recall\": 0.63883847,\n",
    "          \"precision\": 0.5623003,\n",
    "          \"f1Score\": 0.5981308,\n",
    "          \"falsePositiveRate\": 0.06702544,\n",
    "          \"truePositiveCount\": \"352\",\n",
    "          \"falsePositiveCount\": \"274\",\n",
    "          \"falseNegativeCount\": \"199\",\n",
    "          \"trueNegativeCount\": \"3814\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.36666742,\n",
    "          \"recall\": 0.6297641,\n",
    "          \"precision\": 0.56978655,\n",
    "          \"f1Score\": 0.59827584,\n",
    "          \"falsePositiveRate\": 0.06409002,\n",
    "          \"truePositiveCount\": \"347\",\n",
    "          \"falsePositiveCount\": \"262\",\n",
    "          \"falseNegativeCount\": \"204\",\n",
    "          \"trueNegativeCount\": \"3826\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.37787727,\n",
    "          \"recall\": 0.6188748,\n",
    "          \"precision\": 0.5809199,\n",
    "          \"f1Score\": 0.599297,\n",
    "          \"falsePositiveRate\": 0.060176127,\n",
    "          \"truePositiveCount\": \"341\",\n",
    "          \"falsePositiveCount\": \"246\",\n",
    "          \"falseNegativeCount\": \"210\",\n",
    "          \"trueNegativeCount\": \"3842\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39064476,\n",
    "          \"recall\": 0.60980034,\n",
    "          \"precision\": 0.58638746,\n",
    "          \"f1Score\": 0.59786475,\n",
    "          \"falsePositiveRate\": 0.05797456,\n",
    "          \"truePositiveCount\": \"336\",\n",
    "          \"falsePositiveCount\": \"237\",\n",
    "          \"falseNegativeCount\": \"215\",\n",
    "          \"trueNegativeCount\": \"3851\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39657614,\n",
    "          \"recall\": 0.59891105,\n",
    "          \"precision\": 0.5871886,\n",
    "          \"f1Score\": 0.5929919,\n",
    "          \"falsePositiveRate\": 0.056751467,\n",
    "          \"truePositiveCount\": \"330\",\n",
    "          \"falsePositiveCount\": \"232\",\n",
    "          \"falseNegativeCount\": \"221\",\n",
    "          \"trueNegativeCount\": \"3856\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39991632,\n",
    "          \"recall\": 0.58983666,\n",
    "          \"precision\": 0.5866426,\n",
    "          \"f1Score\": 0.5882353,\n",
    "          \"falsePositiveRate\": 0.05601761,\n",
    "          \"truePositiveCount\": \"325\",\n",
    "          \"falsePositiveCount\": \"229\",\n",
    "          \"falseNegativeCount\": \"226\",\n",
    "          \"trueNegativeCount\": \"3859\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.41141164,\n",
    "          \"recall\": 0.57894737,\n",
    "          \"precision\": 0.5896488,\n",
    "          \"f1Score\": 0.5842491,\n",
    "          \"falsePositiveRate\": 0.054305285,\n",
    "          \"truePositiveCount\": \"319\",\n",
    "          \"falsePositiveCount\": \"222\",\n",
    "          \"falseNegativeCount\": \"232\",\n",
    "          \"trueNegativeCount\": \"3866\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.4184669,\n",
    "          \"recall\": 0.569873,\n",
    "          \"precision\": 0.5902256,\n",
    "          \"f1Score\": 0.5798707,\n",
    "          \"falsePositiveRate\": 0.05332681,\n",
    "          \"truePositiveCount\": \"314\",\n",
    "          \"falsePositiveCount\": \"218\",\n",
    "          \"falseNegativeCount\": \"237\",\n",
    "          \"trueNegativeCount\": \"3870\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.42953506,\n",
    "          \"recall\": 0.5589837,\n",
    "          \"precision\": 0.5980582,\n",
    "          \"f1Score\": 0.5778612,\n",
    "          \"falsePositiveRate\": 0.05063601,\n",
    "          \"truePositiveCount\": \"308\",\n",
    "          \"falsePositiveCount\": \"207\",\n",
    "          \"falseNegativeCount\": \"243\",\n",
    "          \"trueNegativeCount\": \"3881\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.43759817,\n",
    "          \"recall\": 0.54990923,\n",
    "          \"precision\": 0.6023857,\n",
    "          \"f1Score\": 0.57495254,\n",
    "          \"falsePositiveRate\": 0.04892368,\n",
    "          \"truePositiveCount\": \"303\",\n",
    "          \"falsePositiveCount\": \"200\",\n",
    "          \"falseNegativeCount\": \"248\",\n",
    "          \"trueNegativeCount\": \"3888\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.44597256,\n",
    "          \"recall\": 0.53901994,\n",
    "          \"precision\": 0.60365856,\n",
    "          \"f1Score\": 0.569511,\n",
    "          \"falsePositiveRate\": 0.047700588,\n",
    "          \"truePositiveCount\": \"297\",\n",
    "          \"falsePositiveCount\": \"195\",\n",
    "          \"falseNegativeCount\": \"254\",\n",
    "          \"trueNegativeCount\": \"3893\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.44895133,\n",
    "          \"recall\": 0.52994555,\n",
    "          \"precision\": 0.6045549,\n",
    "          \"f1Score\": 0.5647969,\n",
    "          \"falsePositiveRate\": 0.046722114,\n",
    "          \"truePositiveCount\": \"292\",\n",
    "          \"falsePositiveCount\": \"191\",\n",
    "          \"falseNegativeCount\": \"259\",\n",
    "          \"trueNegativeCount\": \"3897\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.46139872,\n",
    "          \"recall\": 0.51905626,\n",
    "          \"precision\": 0.6085106,\n",
    "          \"f1Score\": 0.5602351,\n",
    "          \"falsePositiveRate\": 0.045009784,\n",
    "          \"truePositiveCount\": \"286\",\n",
    "          \"falsePositiveCount\": \"184\",\n",
    "          \"falseNegativeCount\": \"265\",\n",
    "          \"trueNegativeCount\": \"3904\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.47061405,\n",
    "          \"recall\": 0.5099819,\n",
    "          \"precision\": 0.61622804,\n",
    "          \"f1Score\": 0.55809337,\n",
    "          \"falsePositiveRate\": 0.04280822,\n",
    "          \"truePositiveCount\": \"281\",\n",
    "          \"falsePositiveCount\": \"175\",\n",
    "          \"falseNegativeCount\": \"270\",\n",
    "          \"trueNegativeCount\": \"3913\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.48159525,\n",
    "          \"recall\": 0.49909255,\n",
    "          \"precision\": 0.6207675,\n",
    "          \"f1Score\": 0.55331993,\n",
    "          \"falsePositiveRate\": 0.04109589,\n",
    "          \"truePositiveCount\": \"275\",\n",
    "          \"falsePositiveCount\": \"168\",\n",
    "          \"falseNegativeCount\": \"276\",\n",
    "          \"trueNegativeCount\": \"3920\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.48736593,\n",
    "          \"recall\": 0.48820326,\n",
    "          \"precision\": 0.6226852,\n",
    "          \"f1Score\": 0.54730415,\n",
    "          \"falsePositiveRate\": 0.0398728,\n",
    "          \"truePositiveCount\": \"269\",\n",
    "          \"falsePositiveCount\": \"163\",\n",
    "          \"falseNegativeCount\": \"282\",\n",
    "          \"trueNegativeCount\": \"3925\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.4939781,\n",
    "          \"recall\": 0.47912887,\n",
    "          \"precision\": 0.6300716,\n",
    "          \"f1Score\": 0.5443299,\n",
    "          \"falsePositiveRate\": 0.037915852,\n",
    "          \"truePositiveCount\": \"264\",\n",
    "          \"falsePositiveCount\": \"155\",\n",
    "          \"falseNegativeCount\": \"287\",\n",
    "          \"trueNegativeCount\": \"3933\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.4995014,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.635468,\n",
    "          \"f1Score\": 0.5391849,\n",
    "          \"falsePositiveRate\": 0.036203522,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"148\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3940\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5025096,\n",
    "          \"recall\": 0.45916516,\n",
    "          \"precision\": 0.63408524,\n",
    "          \"f1Score\": 0.5326316,\n",
    "          \"falsePositiveRate\": 0.035714287,\n",
    "          \"truePositiveCount\": \"253\",\n",
    "          \"falsePositiveCount\": \"146\",\n",
    "          \"falseNegativeCount\": \"298\",\n",
    "          \"trueNegativeCount\": \"3942\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51137525,\n",
    "          \"recall\": 0.44827586,\n",
    "          \"precision\": 0.63010204,\n",
    "          \"f1Score\": 0.52386004,\n",
    "          \"falsePositiveRate\": 0.035469666,\n",
    "          \"truePositiveCount\": \"247\",\n",
    "          \"falsePositiveCount\": \"145\",\n",
    "          \"falseNegativeCount\": \"304\",\n",
    "          \"trueNegativeCount\": \"3943\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51734245,\n",
    "          \"recall\": 0.43920144,\n",
    "          \"precision\": 0.62857145,\n",
    "          \"f1Score\": 0.517094,\n",
    "          \"falsePositiveRate\": 0.03498043,\n",
    "          \"truePositiveCount\": \"242\",\n",
    "          \"falsePositiveCount\": \"143\",\n",
    "          \"falseNegativeCount\": \"309\",\n",
    "          \"trueNegativeCount\": \"3945\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5248498,\n",
    "          \"recall\": 0.42831215,\n",
    "          \"precision\": 0.631016,\n",
    "          \"f1Score\": 0.5102703,\n",
    "          \"falsePositiveRate\": 0.03375734,\n",
    "          \"truePositiveCount\": \"236\",\n",
    "          \"falsePositiveCount\": \"138\",\n",
    "          \"falseNegativeCount\": \"315\",\n",
    "          \"trueNegativeCount\": \"3950\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.53241414,\n",
    "          \"recall\": 0.41923776,\n",
    "          \"precision\": 0.64705884,\n",
    "          \"f1Score\": 0.5088106,\n",
    "          \"falsePositiveRate\": 0.030821918,\n",
    "          \"truePositiveCount\": \"231\",\n",
    "          \"falsePositiveCount\": \"126\",\n",
    "          \"falseNegativeCount\": \"320\",\n",
    "          \"trueNegativeCount\": \"3962\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5427009,\n",
    "          \"recall\": 0.40834847,\n",
    "          \"precision\": 0.64841497,\n",
    "          \"f1Score\": 0.5011136,\n",
    "          \"falsePositiveRate\": 0.029843444,\n",
    "          \"truePositiveCount\": \"225\",\n",
    "          \"falsePositiveCount\": \"122\",\n",
    "          \"falseNegativeCount\": \"326\",\n",
    "          \"trueNegativeCount\": \"3966\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5505563,\n",
    "          \"recall\": 0.39927405,\n",
    "          \"precision\": 0.6547619,\n",
    "          \"f1Score\": 0.4960541,\n",
    "          \"falsePositiveRate\": 0.028375734,\n",
    "          \"truePositiveCount\": \"220\",\n",
    "          \"falsePositiveCount\": \"116\",\n",
    "          \"falseNegativeCount\": \"331\",\n",
    "          \"trueNegativeCount\": \"3972\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.55334836,\n",
    "          \"recall\": 0.38838476,\n",
    "          \"precision\": 0.65045595,\n",
    "          \"f1Score\": 0.48636365,\n",
    "          \"falsePositiveRate\": 0.028131116,\n",
    "          \"truePositiveCount\": \"214\",\n",
    "          \"falsePositiveCount\": \"115\",\n",
    "          \"falseNegativeCount\": \"337\",\n",
    "          \"trueNegativeCount\": \"3973\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.55837405,\n",
    "          \"recall\": 0.37931034,\n",
    "          \"precision\": 0.6490683,\n",
    "          \"f1Score\": 0.4788087,\n",
    "          \"falsePositiveRate\": 0.02764188,\n",
    "          \"truePositiveCount\": \"209\",\n",
    "          \"falsePositiveCount\": \"113\",\n",
    "          \"falseNegativeCount\": \"342\",\n",
    "          \"trueNegativeCount\": \"3975\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5685674,\n",
    "          \"recall\": 0.36842105,\n",
    "          \"precision\": 0.650641,\n",
    "          \"f1Score\": 0.47045192,\n",
    "          \"falsePositiveRate\": 0.026663406,\n",
    "          \"truePositiveCount\": \"203\",\n",
    "          \"falsePositiveCount\": \"109\",\n",
    "          \"falseNegativeCount\": \"348\",\n",
    "          \"trueNegativeCount\": \"3979\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5732897,\n",
    "          \"recall\": 0.35934663,\n",
    "          \"precision\": 0.6578073,\n",
    "          \"f1Score\": 0.46478873,\n",
    "          \"falsePositiveRate\": 0.025195695,\n",
    "          \"truePositiveCount\": \"198\",\n",
    "          \"falsePositiveCount\": \"103\",\n",
    "          \"falseNegativeCount\": \"353\",\n",
    "          \"trueNegativeCount\": \"3985\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5798386,\n",
    "          \"recall\": 0.34845734,\n",
    "          \"precision\": 0.65753424,\n",
    "          \"f1Score\": 0.455516,\n",
    "          \"falsePositiveRate\": 0.02446184,\n",
    "          \"truePositiveCount\": \"192\",\n",
    "          \"falsePositiveCount\": \"100\",\n",
    "          \"falseNegativeCount\": \"359\",\n",
    "          \"trueNegativeCount\": \"3988\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5841479,\n",
    "          \"recall\": 0.33938295,\n",
    "          \"precision\": 0.6561403,\n",
    "          \"f1Score\": 0.4473684,\n",
    "          \"falsePositiveRate\": 0.023972603,\n",
    "          \"truePositiveCount\": \"187\",\n",
    "          \"falsePositiveCount\": \"98\",\n",
    "          \"falseNegativeCount\": \"364\",\n",
    "          \"trueNegativeCount\": \"3990\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.58893955,\n",
    "          \"recall\": 0.32849365,\n",
    "          \"precision\": 0.6630037,\n",
    "          \"f1Score\": 0.4393204,\n",
    "          \"falsePositiveRate\": 0.022504892,\n",
    "          \"truePositiveCount\": \"181\",\n",
    "          \"falsePositiveCount\": \"92\",\n",
    "          \"falseNegativeCount\": \"370\",\n",
    "          \"trueNegativeCount\": \"3996\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.59818196,\n",
    "          \"recall\": 0.31941923,\n",
    "          \"precision\": 0.6743295,\n",
    "          \"f1Score\": 0.43349755,\n",
    "          \"falsePositiveRate\": 0.020792564,\n",
    "          \"truePositiveCount\": \"176\",\n",
    "          \"falsePositiveCount\": \"85\",\n",
    "          \"falseNegativeCount\": \"375\",\n",
    "          \"trueNegativeCount\": \"4003\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.60704446,\n",
    "          \"recall\": 0.30852994,\n",
    "          \"precision\": 0.6854839,\n",
    "          \"f1Score\": 0.42553192,\n",
    "          \"falsePositiveRate\": 0.019080235,\n",
    "          \"truePositiveCount\": \"170\",\n",
    "          \"falsePositiveCount\": \"78\",\n",
    "          \"falseNegativeCount\": \"381\",\n",
    "          \"trueNegativeCount\": \"4010\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6096991,\n",
    "          \"recall\": 0.29945552,\n",
    "          \"precision\": 0.6818182,\n",
    "          \"f1Score\": 0.41614124,\n",
    "          \"falsePositiveRate\": 0.018835617,\n",
    "          \"truePositiveCount\": \"165\",\n",
    "          \"falsePositiveCount\": \"77\",\n",
    "          \"falseNegativeCount\": \"386\",\n",
    "          \"trueNegativeCount\": \"4011\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6194289,\n",
    "          \"recall\": 0.28856623,\n",
    "          \"precision\": 0.6913043,\n",
    "          \"f1Score\": 0.4071703,\n",
    "          \"falsePositiveRate\": 0.017367907,\n",
    "          \"truePositiveCount\": \"159\",\n",
    "          \"falsePositiveCount\": \"71\",\n",
    "          \"falseNegativeCount\": \"392\",\n",
    "          \"trueNegativeCount\": \"4017\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6299003,\n",
    "          \"recall\": 0.27949184,\n",
    "          \"precision\": 0.6936937,\n",
    "          \"f1Score\": 0.3984476,\n",
    "          \"falsePositiveRate\": 0.01663405,\n",
    "          \"truePositiveCount\": \"154\",\n",
    "          \"falsePositiveCount\": \"68\",\n",
    "          \"falseNegativeCount\": \"397\",\n",
    "          \"trueNegativeCount\": \"4020\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6326805,\n",
    "          \"recall\": 0.26860255,\n",
    "          \"precision\": 0.6883721,\n",
    "          \"f1Score\": 0.38642296,\n",
    "          \"falsePositiveRate\": 0.016389433,\n",
    "          \"truePositiveCount\": \"148\",\n",
    "          \"falsePositiveCount\": \"67\",\n",
    "          \"falseNegativeCount\": \"403\",\n",
    "          \"trueNegativeCount\": \"4021\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6446329,\n",
    "          \"recall\": 0.25952813,\n",
    "          \"precision\": 0.71859294,\n",
    "          \"f1Score\": 0.38133332,\n",
    "          \"falsePositiveRate\": 0.01369863,\n",
    "          \"truePositiveCount\": \"143\",\n",
    "          \"falsePositiveCount\": \"56\",\n",
    "          \"falseNegativeCount\": \"408\",\n",
    "          \"trueNegativeCount\": \"4032\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6525863,\n",
    "          \"recall\": 0.24863884,\n",
    "          \"precision\": 0.7287234,\n",
    "          \"f1Score\": 0.37077132,\n",
    "          \"falsePositiveRate\": 0.012475538,\n",
    "          \"truePositiveCount\": \"137\",\n",
    "          \"falsePositiveCount\": \"51\",\n",
    "          \"falseNegativeCount\": \"414\",\n",
    "          \"trueNegativeCount\": \"4037\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6613302,\n",
    "          \"recall\": 0.23956443,\n",
    "          \"precision\": 0.73333335,\n",
    "          \"f1Score\": 0.3611491,\n",
    "          \"falsePositiveRate\": 0.011741683,\n",
    "          \"truePositiveCount\": \"132\",\n",
    "          \"falsePositiveCount\": \"48\",\n",
    "          \"falseNegativeCount\": \"419\",\n",
    "          \"trueNegativeCount\": \"4040\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6681088,\n",
    "          \"recall\": 0.22867514,\n",
    "          \"precision\": 0.73255813,\n",
    "          \"f1Score\": 0.34854773,\n",
    "          \"falsePositiveRate\": 0.011252446,\n",
    "          \"truePositiveCount\": \"126\",\n",
    "          \"falsePositiveCount\": \"46\",\n",
    "          \"falseNegativeCount\": \"425\",\n",
    "          \"trueNegativeCount\": \"4042\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6787454,\n",
    "          \"recall\": 0.21960072,\n",
    "          \"precision\": 0.7378049,\n",
    "          \"f1Score\": 0.33846155,\n",
    "          \"falsePositiveRate\": 0.010518591,\n",
    "          \"truePositiveCount\": \"121\",\n",
    "          \"falsePositiveCount\": \"43\",\n",
    "          \"falseNegativeCount\": \"430\",\n",
    "          \"trueNegativeCount\": \"4045\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.68742466,\n",
    "          \"recall\": 0.20871143,\n",
    "          \"precision\": 0.73717946,\n",
    "          \"f1Score\": 0.32531825,\n",
    "          \"falsePositiveRate\": 0.010029354,\n",
    "          \"truePositiveCount\": \"115\",\n",
    "          \"falsePositiveCount\": \"41\",\n",
    "          \"falseNegativeCount\": \"436\",\n",
    "          \"trueNegativeCount\": \"4047\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.69688004,\n",
    "          \"recall\": 0.19963703,\n",
    "          \"precision\": 0.7482993,\n",
    "          \"f1Score\": 0.31518623,\n",
    "          \"falsePositiveRate\": 0.009050881,\n",
    "          \"truePositiveCount\": \"110\",\n",
    "          \"falsePositiveCount\": \"37\",\n",
    "          \"falseNegativeCount\": \"441\",\n",
    "          \"trueNegativeCount\": \"4051\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.70484525,\n",
    "          \"recall\": 0.18874773,\n",
    "          \"precision\": 0.7647059,\n",
    "          \"f1Score\": 0.30276564,\n",
    "          \"falsePositiveRate\": 0.007827789,\n",
    "          \"truePositiveCount\": \"104\",\n",
    "          \"falsePositiveCount\": \"32\",\n",
    "          \"falseNegativeCount\": \"447\",\n",
    "          \"trueNegativeCount\": \"4056\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.71349317,\n",
    "          \"recall\": 0.17967331,\n",
    "          \"precision\": 0.76153845,\n",
    "          \"f1Score\": 0.2907489,\n",
    "          \"falsePositiveRate\": 0.00758317,\n",
    "          \"truePositiveCount\": \"99\",\n",
    "          \"falsePositiveCount\": \"31\",\n",
    "          \"falseNegativeCount\": \"452\",\n",
    "          \"trueNegativeCount\": \"4057\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.72145003,\n",
    "          \"recall\": 0.16878402,\n",
    "          \"precision\": 0.75609756,\n",
    "          \"f1Score\": 0.27596438,\n",
    "          \"falsePositiveRate\": 0.007338552,\n",
    "          \"truePositiveCount\": \"93\",\n",
    "          \"falsePositiveCount\": \"30\",\n",
    "          \"falseNegativeCount\": \"458\",\n",
    "          \"trueNegativeCount\": \"4058\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.72871804,\n",
    "          \"recall\": 0.15970962,\n",
    "          \"precision\": 0.76521736,\n",
    "          \"f1Score\": 0.26426426,\n",
    "          \"falsePositiveRate\": 0.0066046966,\n",
    "          \"truePositiveCount\": \"88\",\n",
    "          \"falsePositiveCount\": \"27\",\n",
    "          \"falseNegativeCount\": \"463\",\n",
    "          \"trueNegativeCount\": \"4061\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.74161375,\n",
    "          \"recall\": 0.14882033,\n",
    "          \"precision\": 0.76635516,\n",
    "          \"f1Score\": 0.24924012,\n",
    "          \"falsePositiveRate\": 0.00611546,\n",
    "          \"truePositiveCount\": \"82\",\n",
    "          \"falsePositiveCount\": \"25\",\n",
    "          \"falseNegativeCount\": \"469\",\n",
    "          \"trueNegativeCount\": \"4063\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.74766874,\n",
    "          \"recall\": 0.13974592,\n",
    "          \"precision\": 0.78571427,\n",
    "          \"f1Score\": 0.23728813,\n",
    "          \"falsePositiveRate\": 0.0051369863,\n",
    "          \"truePositiveCount\": \"77\",\n",
    "          \"falsePositiveCount\": \"21\",\n",
    "          \"falseNegativeCount\": \"474\",\n",
    "          \"trueNegativeCount\": \"4067\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.75845456,\n",
    "          \"recall\": 0.12885663,\n",
    "          \"precision\": 0.8068182,\n",
    "          \"f1Score\": 0.22222222,\n",
    "          \"falsePositiveRate\": 0.0041585127,\n",
    "          \"truePositiveCount\": \"71\",\n",
    "          \"falsePositiveCount\": \"17\",\n",
    "          \"falseNegativeCount\": \"480\",\n",
    "          \"trueNegativeCount\": \"4071\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7666716,\n",
    "          \"recall\": 0.11978222,\n",
    "          \"precision\": 0.80487806,\n",
    "          \"f1Score\": 0.2085308,\n",
    "          \"falsePositiveRate\": 0.0039138943,\n",
    "          \"truePositiveCount\": \"66\",\n",
    "          \"falsePositiveCount\": \"16\",\n",
    "          \"falseNegativeCount\": \"485\",\n",
    "          \"trueNegativeCount\": \"4072\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7772814,\n",
    "          \"recall\": 0.108892925,\n",
    "          \"precision\": 0.7894737,\n",
    "          \"f1Score\": 0.19138756,\n",
    "          \"falsePositiveRate\": 0.0039138943,\n",
    "          \"truePositiveCount\": \"60\",\n",
    "          \"falsePositiveCount\": \"16\",\n",
    "          \"falseNegativeCount\": \"491\",\n",
    "          \"trueNegativeCount\": \"4072\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.78661436,\n",
    "          \"recall\": 0.09981851,\n",
    "          \"precision\": 0.8088235,\n",
    "          \"f1Score\": 0.17770597,\n",
    "          \"falsePositiveRate\": 0.003180039,\n",
    "          \"truePositiveCount\": \"55\",\n",
    "          \"falsePositiveCount\": \"13\",\n",
    "          \"falseNegativeCount\": \"496\",\n",
    "          \"trueNegativeCount\": \"4075\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8050728,\n",
    "          \"recall\": 0.08892922,\n",
    "          \"precision\": 0.8448276,\n",
    "          \"f1Score\": 0.16091955,\n",
    "          \"falsePositiveRate\": 0.0022015655,\n",
    "          \"truePositiveCount\": \"49\",\n",
    "          \"falsePositiveCount\": \"9\",\n",
    "          \"falseNegativeCount\": \"502\",\n",
    "          \"trueNegativeCount\": \"4079\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8127946,\n",
    "          \"recall\": 0.07985481,\n",
    "          \"precision\": 0.8301887,\n",
    "          \"f1Score\": 0.14569536,\n",
    "          \"falsePositiveRate\": 0.0022015655,\n",
    "          \"truePositiveCount\": \"44\",\n",
    "          \"falsePositiveCount\": \"9\",\n",
    "          \"falseNegativeCount\": \"507\",\n",
    "          \"trueNegativeCount\": \"4079\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.82794523,\n",
    "          \"recall\": 0.06896552,\n",
    "          \"precision\": 0.82608694,\n",
    "          \"f1Score\": 0.12730318,\n",
    "          \"falsePositiveRate\": 0.0019569471,\n",
    "          \"truePositiveCount\": \"38\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"513\",\n",
    "          \"trueNegativeCount\": \"4080\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.83858025,\n",
    "          \"recall\": 0.05989111,\n",
    "          \"precision\": 0.84615386,\n",
    "          \"f1Score\": 0.11186441,\n",
    "          \"falsePositiveRate\": 0.0014677104,\n",
    "          \"truePositiveCount\": \"33\",\n",
    "          \"falsePositiveCount\": \"6\",\n",
    "          \"falseNegativeCount\": \"518\",\n",
    "          \"trueNegativeCount\": \"4082\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.85226774,\n",
    "          \"recall\": 0.049001817,\n",
    "          \"precision\": 0.87096775,\n",
    "          \"f1Score\": 0.0927835,\n",
    "          \"falsePositiveRate\": 0.0009784736,\n",
    "          \"truePositiveCount\": \"27\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"524\",\n",
    "          \"trueNegativeCount\": \"4084\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.86955357,\n",
    "          \"recall\": 0.039927404,\n",
    "          \"precision\": 0.88,\n",
    "          \"f1Score\": 0.07638889,\n",
    "          \"falsePositiveRate\": 0.0007338552,\n",
    "          \"truePositiveCount\": \"22\",\n",
    "          \"falsePositiveCount\": \"3\",\n",
    "          \"falseNegativeCount\": \"529\",\n",
    "          \"trueNegativeCount\": \"4085\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8754055,\n",
    "          \"recall\": 0.029038113,\n",
    "          \"precision\": 0.84210527,\n",
    "          \"f1Score\": 0.056140352,\n",
    "          \"falsePositiveRate\": 0.0007338552,\n",
    "          \"truePositiveCount\": \"16\",\n",
    "          \"falsePositiveCount\": \"3\",\n",
    "          \"falseNegativeCount\": \"535\",\n",
    "          \"trueNegativeCount\": \"4085\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.888999,\n",
    "          \"recall\": 0.019963702,\n",
    "          \"precision\": 0.78571427,\n",
    "          \"f1Score\": 0.038938053,\n",
    "          \"falsePositiveRate\": 0.0007338552,\n",
    "          \"truePositiveCount\": \"11\",\n",
    "          \"falsePositiveCount\": \"3\",\n",
    "          \"falseNegativeCount\": \"540\",\n",
    "          \"trueNegativeCount\": \"4085\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.912919,\n",
    "          \"recall\": 0.00907441,\n",
    "          \"precision\": 0.71428573,\n",
    "          \"f1Score\": 0.017921148,\n",
    "          \"falsePositiveRate\": 0.0004892368,\n",
    "          \"truePositiveCount\": \"5\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"546\",\n",
    "          \"trueNegativeCount\": \"4086\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9486238,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"551\",\n",
    "          \"trueNegativeCount\": \"4088\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 1.0,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"551\",\n",
    "          \"trueNegativeCount\": \"4088\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        }\n",
    "      ],\n",
    "      \"auRoc\": 0.9330749,\n",
    "      \"logLoss\": 0.993795\n",
    "    },\n",
    "    \"displayName\": \"2\"\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/5840134723993193762\",\n",
    "    \"annotationSpecId\": \"not available\",\n",
    "    \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "    \"evaluatedExampleCount\": 4088,\n",
    "    \"classificationEvaluationMetrics\": {\n",
    "      \"auPrc\": 0.9580479,\n",
    "      \"confidenceMetricsEntry\": [\n",
    "        {\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.07797722,\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.30998033,\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39478266,\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.46149504,\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5,\n",
    "          \"recall\": 0.9640411,\n",
    "          \"precision\": 0.9307983,\n",
    "          \"f1Score\": 0.9471281,\n",
    "          \"falsePositiveRate\": 0.53176045,\n",
    "          \"truePositiveCount\": \"3941\",\n",
    "          \"falsePositiveCount\": \"293\",\n",
    "          \"falseNegativeCount\": \"147\",\n",
    "          \"trueNegativeCount\": \"258\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51386535,\n",
    "          \"recall\": 0.95988256,\n",
    "          \"precision\": 0.93361884,\n",
    "          \"f1Score\": 0.94656855,\n",
    "          \"falsePositiveRate\": 0.50635207,\n",
    "          \"truePositiveCount\": \"3924\",\n",
    "          \"falsePositiveCount\": \"279\",\n",
    "          \"falseNegativeCount\": \"164\",\n",
    "          \"trueNegativeCount\": \"272\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5659395,\n",
    "          \"recall\": 0.94985324,\n",
    "          \"precision\": 0.9406492,\n",
    "          \"f1Score\": 0.9452288,\n",
    "          \"falsePositiveRate\": 0.4446461,\n",
    "          \"truePositiveCount\": \"3883\",\n",
    "          \"falsePositiveCount\": \"245\",\n",
    "          \"falseNegativeCount\": \"205\",\n",
    "          \"trueNegativeCount\": \"306\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6215324,\n",
    "          \"recall\": 0.93982387,\n",
    "          \"precision\": 0.94817376,\n",
    "          \"f1Score\": 0.94398034,\n",
    "          \"falsePositiveRate\": 0.38112524,\n",
    "          \"truePositiveCount\": \"3842\",\n",
    "          \"falsePositiveCount\": \"210\",\n",
    "          \"falseNegativeCount\": \"246\",\n",
    "          \"trueNegativeCount\": \"341\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.65884733,\n",
    "          \"recall\": 0.93003917,\n",
    "          \"precision\": 0.9533601,\n",
    "          \"f1Score\": 0.9415552,\n",
    "          \"falsePositiveRate\": 0.33756804,\n",
    "          \"truePositiveCount\": \"3802\",\n",
    "          \"falsePositiveCount\": \"186\",\n",
    "          \"falseNegativeCount\": \"286\",\n",
    "          \"trueNegativeCount\": \"365\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7143411,\n",
    "          \"recall\": 0.9200098,\n",
    "          \"precision\": 0.9623849,\n",
    "          \"f1Score\": 0.9407204,\n",
    "          \"falsePositiveRate\": 0.26678765,\n",
    "          \"truePositiveCount\": \"3761\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"327\",\n",
    "          \"trueNegativeCount\": \"404\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7492718,\n",
    "          \"recall\": 0.9099804,\n",
    "          \"precision\": 0.96749026,\n",
    "          \"f1Score\": 0.9378545,\n",
    "          \"falsePositiveRate\": 0.22686026,\n",
    "          \"truePositiveCount\": \"3720\",\n",
    "          \"falsePositiveCount\": \"125\",\n",
    "          \"falseNegativeCount\": \"368\",\n",
    "          \"trueNegativeCount\": \"426\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7795446,\n",
    "          \"recall\": 0.9001957,\n",
    "          \"precision\": 0.97303015,\n",
    "          \"f1Score\": 0.93519694,\n",
    "          \"falsePositiveRate\": 0.18511796,\n",
    "          \"truePositiveCount\": \"3680\",\n",
    "          \"falsePositiveCount\": \"102\",\n",
    "          \"falseNegativeCount\": \"408\",\n",
    "          \"trueNegativeCount\": \"449\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7989856,\n",
    "          \"recall\": 0.8899217,\n",
    "          \"precision\": 0.97559667,\n",
    "          \"f1Score\": 0.93079185,\n",
    "          \"falsePositiveRate\": 0.16515426,\n",
    "          \"truePositiveCount\": \"3638\",\n",
    "          \"falsePositiveCount\": \"91\",\n",
    "          \"falseNegativeCount\": \"450\",\n",
    "          \"trueNegativeCount\": \"460\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.82042044,\n",
    "          \"recall\": 0.87989235,\n",
    "          \"precision\": 0.9777113,\n",
    "          \"f1Score\": 0.9262263,\n",
    "          \"falsePositiveRate\": 0.14882033,\n",
    "          \"truePositiveCount\": \"3597\",\n",
    "          \"falsePositiveCount\": \"82\",\n",
    "          \"falseNegativeCount\": \"491\",\n",
    "          \"trueNegativeCount\": \"469\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.83562684,\n",
    "          \"recall\": 0.87010765,\n",
    "          \"precision\": 0.9790807,\n",
    "          \"f1Score\": 0.92138326,\n",
    "          \"falsePositiveRate\": 0.13793103,\n",
    "          \"truePositiveCount\": \"3557\",\n",
    "          \"falsePositiveCount\": \"76\",\n",
    "          \"falseNegativeCount\": \"531\",\n",
    "          \"trueNegativeCount\": \"475\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.85593396,\n",
    "          \"recall\": 0.8600783,\n",
    "          \"precision\": 0.98102677,\n",
    "          \"f1Score\": 0.9165798,\n",
    "          \"falsePositiveRate\": 0.123411976,\n",
    "          \"truePositiveCount\": \"3516\",\n",
    "          \"falsePositiveCount\": \"68\",\n",
    "          \"falseNegativeCount\": \"572\",\n",
    "          \"trueNegativeCount\": \"483\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.86959773,\n",
    "          \"recall\": 0.8500489,\n",
    "          \"precision\": 0.9810841,\n",
    "          \"f1Score\": 0.9108781,\n",
    "          \"falsePositiveRate\": 0.1215971,\n",
    "          \"truePositiveCount\": \"3475\",\n",
    "          \"falsePositiveCount\": \"67\",\n",
    "          \"falseNegativeCount\": \"613\",\n",
    "          \"trueNegativeCount\": \"484\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.88389677,\n",
    "          \"recall\": 0.83977497,\n",
    "          \"precision\": 0.98338586,\n",
    "          \"f1Score\": 0.90592426,\n",
    "          \"falsePositiveRate\": 0.10526316,\n",
    "          \"truePositiveCount\": \"3433\",\n",
    "          \"falsePositiveCount\": \"58\",\n",
    "          \"falseNegativeCount\": \"655\",\n",
    "          \"trueNegativeCount\": \"493\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.89513993,\n",
    "          \"recall\": 0.8299902,\n",
    "          \"precision\": 0.9840487,\n",
    "          \"f1Score\": 0.9004777,\n",
    "          \"falsePositiveRate\": 0.09981851,\n",
    "          \"truePositiveCount\": \"3393\",\n",
    "          \"falsePositiveCount\": \"55\",\n",
    "          \"falseNegativeCount\": \"695\",\n",
    "          \"trueNegativeCount\": \"496\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.90273213,\n",
    "          \"recall\": 0.82020545,\n",
    "          \"precision\": 0.9853071,\n",
    "          \"f1Score\": 0.8952076,\n",
    "          \"falsePositiveRate\": 0.0907441,\n",
    "          \"truePositiveCount\": \"3353\",\n",
    "          \"falsePositiveCount\": \"50\",\n",
    "          \"falseNegativeCount\": \"735\",\n",
    "          \"trueNegativeCount\": \"501\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9118437,\n",
    "          \"recall\": 0.81017613,\n",
    "          \"precision\": 0.98718333,\n",
    "          \"f1Score\": 0.88996375,\n",
    "          \"falsePositiveRate\": 0.07803993,\n",
    "          \"truePositiveCount\": \"3312\",\n",
    "          \"falsePositiveCount\": \"43\",\n",
    "          \"falseNegativeCount\": \"776\",\n",
    "          \"trueNegativeCount\": \"508\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9204901,\n",
    "          \"recall\": 0.79990214,\n",
    "          \"precision\": 0.98821396,\n",
    "          \"f1Score\": 0.8841422,\n",
    "          \"falsePositiveRate\": 0.0707804,\n",
    "          \"truePositiveCount\": \"3270\",\n",
    "          \"falsePositiveCount\": \"39\",\n",
    "          \"falseNegativeCount\": \"818\",\n",
    "          \"trueNegativeCount\": \"512\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9283992,\n",
    "          \"recall\": 0.7898728,\n",
    "          \"precision\": 0.9898835,\n",
    "          \"f1Score\": 0.87863946,\n",
    "          \"falsePositiveRate\": 0.05989111,\n",
    "          \"truePositiveCount\": \"3229\",\n",
    "          \"falsePositiveCount\": \"33\",\n",
    "          \"falseNegativeCount\": \"859\",\n",
    "          \"trueNegativeCount\": \"518\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.934707,\n",
    "          \"recall\": 0.77984345,\n",
    "          \"precision\": 0.9909854,\n",
    "          \"f1Score\": 0.8728268,\n",
    "          \"falsePositiveRate\": 0.05263158,\n",
    "          \"truePositiveCount\": \"3188\",\n",
    "          \"falsePositiveCount\": \"29\",\n",
    "          \"falseNegativeCount\": \"900\",\n",
    "          \"trueNegativeCount\": \"522\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9401358,\n",
    "          \"recall\": 0.7698141,\n",
    "          \"precision\": 0.99211854,\n",
    "          \"f1Score\": 0.86694217,\n",
    "          \"falsePositiveRate\": 0.04537205,\n",
    "          \"truePositiveCount\": \"3147\",\n",
    "          \"falsePositiveCount\": \"25\",\n",
    "          \"falseNegativeCount\": \"941\",\n",
    "          \"trueNegativeCount\": \"526\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.943966,\n",
    "          \"recall\": 0.7600294,\n",
    "          \"precision\": 0.9923347,\n",
    "          \"f1Score\": 0.86078405,\n",
    "          \"falsePositiveRate\": 0.043557167,\n",
    "          \"truePositiveCount\": \"3107\",\n",
    "          \"falsePositiveCount\": \"24\",\n",
    "          \"falseNegativeCount\": \"981\",\n",
    "          \"trueNegativeCount\": \"527\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9484665,\n",
    "          \"recall\": 0.7497554,\n",
    "          \"precision\": 0.9925518,\n",
    "          \"f1Score\": 0.85423636,\n",
    "          \"falsePositiveRate\": 0.041742288,\n",
    "          \"truePositiveCount\": \"3065\",\n",
    "          \"falsePositiveCount\": \"23\",\n",
    "          \"falseNegativeCount\": \"1023\",\n",
    "          \"trueNegativeCount\": \"528\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9512129,\n",
    "          \"recall\": 0.7399706,\n",
    "          \"precision\": 0.9931057,\n",
    "          \"f1Score\": 0.8480516,\n",
    "          \"falsePositiveRate\": 0.03811252,\n",
    "          \"truePositiveCount\": \"3025\",\n",
    "          \"falsePositiveCount\": \"21\",\n",
    "          \"falseNegativeCount\": \"1063\",\n",
    "          \"trueNegativeCount\": \"530\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95533496,\n",
    "          \"recall\": 0.7299413,\n",
    "          \"precision\": 0.9946667,\n",
    "          \"f1Score\": 0.8419865,\n",
    "          \"falsePositiveRate\": 0.029038113,\n",
    "          \"truePositiveCount\": \"2984\",\n",
    "          \"falsePositiveCount\": \"16\",\n",
    "          \"falseNegativeCount\": \"1104\",\n",
    "          \"trueNegativeCount\": \"535\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95724624,\n",
    "          \"recall\": 0.72015655,\n",
    "          \"precision\": 0.9956037,\n",
    "          \"f1Score\": 0.8357701,\n",
    "          \"falsePositiveRate\": 0.023593467,\n",
    "          \"truePositiveCount\": \"2944\",\n",
    "          \"falsePositiveCount\": \"13\",\n",
    "          \"falseNegativeCount\": \"1144\",\n",
    "          \"trueNegativeCount\": \"538\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96008635,\n",
    "          \"recall\": 0.7101272,\n",
    "          \"precision\": 0.99588335,\n",
    "          \"f1Score\": 0.82907325,\n",
    "          \"falsePositiveRate\": 0.021778584,\n",
    "          \"truePositiveCount\": \"2903\",\n",
    "          \"falsePositiveCount\": \"12\",\n",
    "          \"falseNegativeCount\": \"1185\",\n",
    "          \"trueNegativeCount\": \"539\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9629221,\n",
    "          \"recall\": 0.70009786,\n",
    "          \"precision\": 0.99617124,\n",
    "          \"f1Score\": 0.82229567,\n",
    "          \"falsePositiveRate\": 0.019963702,\n",
    "          \"truePositiveCount\": \"2862\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1226\",\n",
    "          \"trueNegativeCount\": \"540\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9646139,\n",
    "          \"recall\": 0.6900685,\n",
    "          \"precision\": 0.9961158,\n",
    "          \"f1Score\": 0.8153179,\n",
    "          \"falsePositiveRate\": 0.019963702,\n",
    "          \"truePositiveCount\": \"2821\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1267\",\n",
    "          \"trueNegativeCount\": \"540\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96656674,\n",
    "          \"recall\": 0.67979455,\n",
    "          \"precision\": 0.99605733,\n",
    "          \"f1Score\": 0.8080838,\n",
    "          \"falsePositiveRate\": 0.019963702,\n",
    "          \"truePositiveCount\": \"2779\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1309\",\n",
    "          \"trueNegativeCount\": \"540\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96897614,\n",
    "          \"recall\": 0.6700098,\n",
    "          \"precision\": 0.9963623,\n",
    "          \"f1Score\": 0.8012286,\n",
    "          \"falsePositiveRate\": 0.01814882,\n",
    "          \"truePositiveCount\": \"2739\",\n",
    "          \"falsePositiveCount\": \"10\",\n",
    "          \"falseNegativeCount\": \"1349\",\n",
    "          \"trueNegativeCount\": \"541\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9706901,\n",
    "          \"recall\": 0.6599804,\n",
    "          \"precision\": 0.99630725,\n",
    "          \"f1Score\": 0.79399645,\n",
    "          \"falsePositiveRate\": 0.01814882,\n",
    "          \"truePositiveCount\": \"2698\",\n",
    "          \"falsePositiveCount\": \"10\",\n",
    "          \"falseNegativeCount\": \"1390\",\n",
    "          \"trueNegativeCount\": \"541\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97229564,\n",
    "          \"recall\": 0.6499511,\n",
    "          \"precision\": 0.99699813,\n",
    "          \"f1Score\": 0.7869095,\n",
    "          \"falsePositiveRate\": 0.014519056,\n",
    "          \"truePositiveCount\": \"2657\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"1431\",\n",
    "          \"trueNegativeCount\": \"543\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9738624,\n",
    "          \"recall\": 0.6399217,\n",
    "          \"precision\": 0.9969512,\n",
    "          \"f1Score\": 0.7794994,\n",
    "          \"falsePositiveRate\": 0.014519056,\n",
    "          \"truePositiveCount\": \"2616\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"1472\",\n",
    "          \"trueNegativeCount\": \"543\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97582275,\n",
    "          \"recall\": 0.62989235,\n",
    "          \"precision\": 0.99728894,\n",
    "          \"f1Score\": 0.7721139,\n",
    "          \"falsePositiveRate\": 0.012704174,\n",
    "          \"truePositiveCount\": \"2575\",\n",
    "          \"falsePositiveCount\": \"7\",\n",
    "          \"falseNegativeCount\": \"1513\",\n",
    "          \"trueNegativeCount\": \"544\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9772092,\n",
    "          \"recall\": 0.62010765,\n",
    "          \"precision\": 0.9976387,\n",
    "          \"f1Score\": 0.76482123,\n",
    "          \"falsePositiveRate\": 0.010889292,\n",
    "          \"truePositiveCount\": \"2535\",\n",
    "          \"falsePositiveCount\": \"6\",\n",
    "          \"falseNegativeCount\": \"1553\",\n",
    "          \"trueNegativeCount\": \"545\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9781838,\n",
    "          \"recall\": 0.6100783,\n",
    "          \"precision\": 0.9979992,\n",
    "          \"f1Score\": 0.7572491,\n",
    "          \"falsePositiveRate\": 0.00907441,\n",
    "          \"truePositiveCount\": \"2494\",\n",
    "          \"falsePositiveCount\": \"5\",\n",
    "          \"falseNegativeCount\": \"1594\",\n",
    "          \"trueNegativeCount\": \"546\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9794321,\n",
    "          \"recall\": 0.5998043,\n",
    "          \"precision\": 0.99837136,\n",
    "          \"f1Score\": 0.74938875,\n",
    "          \"falsePositiveRate\": 0.007259528,\n",
    "          \"truePositiveCount\": \"2452\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"1636\",\n",
    "          \"trueNegativeCount\": \"547\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9804444,\n",
    "          \"recall\": 0.5900196,\n",
    "          \"precision\": 0.99834436,\n",
    "          \"f1Score\": 0.74169743,\n",
    "          \"falsePositiveRate\": 0.007259528,\n",
    "          \"truePositiveCount\": \"2412\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"1676\",\n",
    "          \"trueNegativeCount\": \"547\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9813577,\n",
    "          \"recall\": 0.5799902,\n",
    "          \"precision\": 0.9991572,\n",
    "          \"f1Score\": 0.7339421,\n",
    "          \"falsePositiveRate\": 0.003629764,\n",
    "          \"truePositiveCount\": \"2371\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"1717\",\n",
    "          \"trueNegativeCount\": \"549\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9822546,\n",
    "          \"recall\": 0.56996083,\n",
    "          \"precision\": 0.99914235,\n",
    "          \"f1Score\": 0.7258567,\n",
    "          \"falsePositiveRate\": 0.003629764,\n",
    "          \"truePositiveCount\": \"2330\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"1758\",\n",
    "          \"trueNegativeCount\": \"549\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9832565,\n",
    "          \"recall\": 0.5599315,\n",
    "          \"precision\": 0.99912703,\n",
    "          \"f1Score\": 0.71766734,\n",
    "          \"falsePositiveRate\": 0.003629764,\n",
    "          \"truePositiveCount\": \"2289\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"1799\",\n",
    "          \"trueNegativeCount\": \"549\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.983971,\n",
    "          \"recall\": 0.54990214,\n",
    "          \"precision\": 0.99955535,\n",
    "          \"f1Score\": 0.709484,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2248\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1840\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98478603,\n",
    "          \"recall\": 0.54011744,\n",
    "          \"precision\": 0.9995473,\n",
    "          \"f1Score\": 0.7012863,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2208\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1880\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98576885,\n",
    "          \"recall\": 0.52984345,\n",
    "          \"precision\": 0.99953854,\n",
    "          \"f1Score\": 0.6925659,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2166\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1922\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9865271,\n",
    "          \"recall\": 0.5198141,\n",
    "          \"precision\": 0.99952966,\n",
    "          \"f1Score\": 0.6839395,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2125\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"1963\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98722655,\n",
    "          \"recall\": 0.5100294,\n",
    "          \"precision\": 0.9995206,\n",
    "          \"f1Score\": 0.675413,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2085\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2003\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98776835,\n",
    "          \"recall\": 0.49975538,\n",
    "          \"precision\": 0.99951077,\n",
    "          \"f1Score\": 0.66634053,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2043\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2045\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98843795,\n",
    "          \"recall\": 0.49021527,\n",
    "          \"precision\": 0.9995012,\n",
    "          \"f1Score\": 0.657804,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"2004\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2084\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98898214,\n",
    "          \"recall\": 0.4801859,\n",
    "          \"precision\": 0.99949086,\n",
    "          \"f1Score\": 0.64871114,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1963\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2125\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98952276,\n",
    "          \"recall\": 0.46991193,\n",
    "          \"precision\": 0.9994797,\n",
    "          \"f1Score\": 0.63926786,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1921\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2167\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99007314,\n",
    "          \"recall\": 0.4598826,\n",
    "          \"precision\": 0.9994684,\n",
    "          \"f1Score\": 0.62992126,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1880\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2208\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.990457,\n",
    "          \"recall\": 0.45009786,\n",
    "          \"precision\": 0.9994568,\n",
    "          \"f1Score\": 0.620678,\n",
    "          \"falsePositiveRate\": 0.001814882,\n",
    "          \"truePositiveCount\": \"1840\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2248\",\n",
    "          \"trueNegativeCount\": \"550\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99079466,\n",
    "          \"recall\": 0.44006848,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.61117715,\n",
    "          \"truePositiveCount\": \"1799\",\n",
    "          \"falseNegativeCount\": \"2289\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9911339,\n",
    "          \"recall\": 0.43003914,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.60143685,\n",
    "          \"truePositiveCount\": \"1758\",\n",
    "          \"falseNegativeCount\": \"2330\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99141955,\n",
    "          \"recall\": 0.4200098,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.591559,\n",
    "          \"truePositiveCount\": \"1717\",\n",
    "          \"falseNegativeCount\": \"2371\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9917892,\n",
    "          \"recall\": 0.40998042,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5815406,\n",
    "          \"truePositiveCount\": \"1676\",\n",
    "          \"falseNegativeCount\": \"2412\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9921363,\n",
    "          \"recall\": 0.39995107,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.57137865,\n",
    "          \"truePositiveCount\": \"1635\",\n",
    "          \"falseNegativeCount\": \"2453\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99253094,\n",
    "          \"recall\": 0.38992172,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.56107,\n",
    "          \"truePositiveCount\": \"1594\",\n",
    "          \"falseNegativeCount\": \"2494\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9928577,\n",
    "          \"recall\": 0.380137,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5508685,\n",
    "          \"truePositiveCount\": \"1554\",\n",
    "          \"falseNegativeCount\": \"2534\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99313205,\n",
    "          \"recall\": 0.369863,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.54,\n",
    "          \"truePositiveCount\": \"1512\",\n",
    "          \"falseNegativeCount\": \"2576\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9933454,\n",
    "          \"recall\": 0.35983366,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5292319,\n",
    "          \"truePositiveCount\": \"1471\",\n",
    "          \"falseNegativeCount\": \"2617\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99355274,\n",
    "          \"recall\": 0.35004893,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5185722,\n",
    "          \"truePositiveCount\": \"1431\",\n",
    "          \"falseNegativeCount\": \"2657\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99378794,\n",
    "          \"recall\": 0.33977494,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.507212,\n",
    "          \"truePositiveCount\": \"1389\",\n",
    "          \"falseNegativeCount\": \"2699\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9939992,\n",
    "          \"recall\": 0.33023483,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.49650607,\n",
    "          \"truePositiveCount\": \"1350\",\n",
    "          \"falseNegativeCount\": \"2738\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9942213,\n",
    "          \"recall\": 0.31996086,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.48480356,\n",
    "          \"truePositiveCount\": \"1308\",\n",
    "          \"falseNegativeCount\": \"2780\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9944552,\n",
    "          \"recall\": 0.30993152,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.47320262,\n",
    "          \"truePositiveCount\": \"1267\",\n",
    "          \"falseNegativeCount\": \"2821\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9946339,\n",
    "          \"recall\": 0.29990214,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.46142265,\n",
    "          \"truePositiveCount\": \"1226\",\n",
    "          \"falseNegativeCount\": \"2862\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99486357,\n",
    "          \"recall\": 0.2898728,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.44945952,\n",
    "          \"truePositiveCount\": \"1185\",\n",
    "          \"falseNegativeCount\": \"2903\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9950511,\n",
    "          \"recall\": 0.28008807,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.4376075,\n",
    "          \"truePositiveCount\": \"1145\",\n",
    "          \"falseNegativeCount\": \"2943\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9953063,\n",
    "          \"recall\": 0.2698141,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.42496628,\n",
    "          \"truePositiveCount\": \"1103\",\n",
    "          \"falseNegativeCount\": \"2985\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99548787,\n",
    "          \"recall\": 0.26002935,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.4127354,\n",
    "          \"truePositiveCount\": \"1063\",\n",
    "          \"falseNegativeCount\": \"3025\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99564606,\n",
    "          \"recall\": 0.25024462,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.40031305,\n",
    "          \"truePositiveCount\": \"1023\",\n",
    "          \"falseNegativeCount\": \"3065\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9958494,\n",
    "          \"recall\": 0.23997064,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.3870586,\n",
    "          \"truePositiveCount\": \"981\",\n",
    "          \"falseNegativeCount\": \"3107\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9959897,\n",
    "          \"recall\": 0.2299413,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.37390614,\n",
    "          \"truePositiveCount\": \"940\",\n",
    "          \"falseNegativeCount\": \"3148\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9961461,\n",
    "          \"recall\": 0.21991193,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.3605374,\n",
    "          \"truePositiveCount\": \"899\",\n",
    "          \"falseNegativeCount\": \"3189\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99626255,\n",
    "          \"recall\": 0.20988259,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.346947,\n",
    "          \"truePositiveCount\": \"858\",\n",
    "          \"falseNegativeCount\": \"3230\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99635655,\n",
    "          \"recall\": 0.19985323,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.33312947,\n",
    "          \"truePositiveCount\": \"817\",\n",
    "          \"falseNegativeCount\": \"3271\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9964815,\n",
    "          \"recall\": 0.1900685,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.31942445,\n",
    "          \"truePositiveCount\": \"777\",\n",
    "          \"falseNegativeCount\": \"3311\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9966108,\n",
    "          \"recall\": 0.17979452,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.30478954,\n",
    "          \"truePositiveCount\": \"735\",\n",
    "          \"falseNegativeCount\": \"3353\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99673027,\n",
    "          \"recall\": 0.16976516,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.29025513,\n",
    "          \"truePositiveCount\": \"694\",\n",
    "          \"falseNegativeCount\": \"3394\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99681914,\n",
    "          \"recall\": 0.15998043,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.27583298,\n",
    "          \"truePositiveCount\": \"654\",\n",
    "          \"falseNegativeCount\": \"3434\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9969455,\n",
    "          \"recall\": 0.15019569,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.26116547,\n",
    "          \"truePositiveCount\": \"614\",\n",
    "          \"falseNegativeCount\": \"3474\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99705213,\n",
    "          \"recall\": 0.13992172,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.24549356,\n",
    "          \"truePositiveCount\": \"572\",\n",
    "          \"falseNegativeCount\": \"3516\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9971576,\n",
    "          \"recall\": 0.13013698,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.23030303,\n",
    "          \"truePositiveCount\": \"532\",\n",
    "          \"falseNegativeCount\": \"3556\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9972461,\n",
    "          \"recall\": 0.11986301,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.21406728,\n",
    "          \"truePositiveCount\": \"490\",\n",
    "          \"falseNegativeCount\": \"3598\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9973425,\n",
    "          \"recall\": 0.110078275,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.19832525,\n",
    "          \"truePositiveCount\": \"450\",\n",
    "          \"falseNegativeCount\": \"3638\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9974355,\n",
    "          \"recall\": 0.10004892,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.18189904,\n",
    "          \"truePositiveCount\": \"409\",\n",
    "          \"falseNegativeCount\": \"3679\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9975045,\n",
    "          \"recall\": 0.09001957,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.16517055,\n",
    "          \"truePositiveCount\": \"368\",\n",
    "          \"falseNegativeCount\": \"3720\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99760437,\n",
    "          \"recall\": 0.079990216,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.14813137,\n",
    "          \"truePositiveCount\": \"327\",\n",
    "          \"falseNegativeCount\": \"3761\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9976864,\n",
    "          \"recall\": 0.07020548,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.1312,\n",
    "          \"truePositiveCount\": \"287\",\n",
    "          \"falseNegativeCount\": \"3801\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9977548,\n",
    "          \"recall\": 0.059931505,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.11308562,\n",
    "          \"truePositiveCount\": \"245\",\n",
    "          \"falseNegativeCount\": \"3843\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99784184,\n",
    "          \"recall\": 0.05014677,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.095504306,\n",
    "          \"truePositiveCount\": \"205\",\n",
    "          \"falseNegativeCount\": \"3883\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9979226,\n",
    "          \"recall\": 0.040117417,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.07714017,\n",
    "          \"truePositiveCount\": \"164\",\n",
    "          \"falseNegativeCount\": \"3924\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9980229,\n",
    "          \"recall\": 0.030088063,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.058418427,\n",
    "          \"truePositiveCount\": \"123\",\n",
    "          \"falseNegativeCount\": \"3965\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9981275,\n",
    "          \"recall\": 0.01981409,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.03885824,\n",
    "          \"truePositiveCount\": \"81\",\n",
    "          \"falseNegativeCount\": \"4007\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9982374,\n",
    "          \"recall\": 0.009784736,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.019379845,\n",
    "          \"truePositiveCount\": \"40\",\n",
    "          \"falseNegativeCount\": \"4048\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9987167,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4088\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 1.0,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4088\",\n",
    "          \"trueNegativeCount\": \"551\",\n",
    "          \"positionThreshold\": 1\n",
    "        }\n",
    "      ],\n",
    "      \"auRoc\": 0.9325568,\n",
    "      \"logLoss\": 0.09388557\n",
    "    },\n",
    "    \"displayName\": \"1\"\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/6226491855750944430\",\n",
    "    \"annotationSpecId\": \"not available\",\n",
    "    \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "    \"evaluatedExampleCount\": 551,\n",
    "    \"classificationEvaluationMetrics\": {\n",
    "      \"auPrc\": 0.34112322,\n",
    "      \"confidenceMetricsEntry\": [\n",
    "        {\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.009267787,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.022539802,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.03954769,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.045538254,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.050852664,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.063637815,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.07210142,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.07750216,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.09185968,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.09674809,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.10816234,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.12236822,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.1267207,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.15273023,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.1649416,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.18867286,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.19824302,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.20855853,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2164171,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2228882,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2343938,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.24437484,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.24899487,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.25527748,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2643137,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.27015588,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.2785622,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.29147997,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.30125153,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.30796623,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.32498023,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.33223236,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.3348615,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.33746937,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.34359238,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.3495217,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.35605764,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.36666742,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.37787727,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39064476,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39657614,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.39991632,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.41141164,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.4184669,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.42953506,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.43759817,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.44597256,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.44895133,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.46139872,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.47061405,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.48159525,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.48736593,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.4939781,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.4995014,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5,\n",
    "          \"recall\": 0.46823958,\n",
    "          \"precision\": 0.63703704,\n",
    "          \"f1Score\": 0.53974897,\n",
    "          \"falsePositiveRate\": 0.035958905,\n",
    "          \"truePositiveCount\": \"258\",\n",
    "          \"falsePositiveCount\": \"147\",\n",
    "          \"falseNegativeCount\": \"293\",\n",
    "          \"trueNegativeCount\": \"3941\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5025096,\n",
    "          \"recall\": 0.45916516,\n",
    "          \"precision\": 0.63408524,\n",
    "          \"f1Score\": 0.5326316,\n",
    "          \"falsePositiveRate\": 0.035714287,\n",
    "          \"truePositiveCount\": \"253\",\n",
    "          \"falsePositiveCount\": \"146\",\n",
    "          \"falseNegativeCount\": \"298\",\n",
    "          \"trueNegativeCount\": \"3942\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51137525,\n",
    "          \"recall\": 0.44827586,\n",
    "          \"precision\": 0.63010204,\n",
    "          \"f1Score\": 0.52386004,\n",
    "          \"falsePositiveRate\": 0.035469666,\n",
    "          \"truePositiveCount\": \"247\",\n",
    "          \"falsePositiveCount\": \"145\",\n",
    "          \"falseNegativeCount\": \"304\",\n",
    "          \"trueNegativeCount\": \"3943\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51734245,\n",
    "          \"recall\": 0.43920144,\n",
    "          \"precision\": 0.62857145,\n",
    "          \"f1Score\": 0.517094,\n",
    "          \"falsePositiveRate\": 0.03498043,\n",
    "          \"truePositiveCount\": \"242\",\n",
    "          \"falsePositiveCount\": \"143\",\n",
    "          \"falseNegativeCount\": \"309\",\n",
    "          \"trueNegativeCount\": \"3945\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5248498,\n",
    "          \"recall\": 0.42831215,\n",
    "          \"precision\": 0.631016,\n",
    "          \"f1Score\": 0.5102703,\n",
    "          \"falsePositiveRate\": 0.03375734,\n",
    "          \"truePositiveCount\": \"236\",\n",
    "          \"falsePositiveCount\": \"138\",\n",
    "          \"falseNegativeCount\": \"315\",\n",
    "          \"trueNegativeCount\": \"3950\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.53241414,\n",
    "          \"recall\": 0.41923776,\n",
    "          \"precision\": 0.64705884,\n",
    "          \"f1Score\": 0.5088106,\n",
    "          \"falsePositiveRate\": 0.030821918,\n",
    "          \"truePositiveCount\": \"231\",\n",
    "          \"falsePositiveCount\": \"126\",\n",
    "          \"falseNegativeCount\": \"320\",\n",
    "          \"trueNegativeCount\": \"3962\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5427009,\n",
    "          \"recall\": 0.40834847,\n",
    "          \"precision\": 0.64841497,\n",
    "          \"f1Score\": 0.5011136,\n",
    "          \"falsePositiveRate\": 0.029843444,\n",
    "          \"truePositiveCount\": \"225\",\n",
    "          \"falsePositiveCount\": \"122\",\n",
    "          \"falseNegativeCount\": \"326\",\n",
    "          \"trueNegativeCount\": \"3966\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5505563,\n",
    "          \"recall\": 0.39927405,\n",
    "          \"precision\": 0.6547619,\n",
    "          \"f1Score\": 0.4960541,\n",
    "          \"falsePositiveRate\": 0.028375734,\n",
    "          \"truePositiveCount\": \"220\",\n",
    "          \"falsePositiveCount\": \"116\",\n",
    "          \"falseNegativeCount\": \"331\",\n",
    "          \"trueNegativeCount\": \"3972\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.55334836,\n",
    "          \"recall\": 0.38838476,\n",
    "          \"precision\": 0.65045595,\n",
    "          \"f1Score\": 0.48636365,\n",
    "          \"falsePositiveRate\": 0.028131116,\n",
    "          \"truePositiveCount\": \"214\",\n",
    "          \"falsePositiveCount\": \"115\",\n",
    "          \"falseNegativeCount\": \"337\",\n",
    "          \"trueNegativeCount\": \"3973\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.55837405,\n",
    "          \"recall\": 0.37931034,\n",
    "          \"precision\": 0.6490683,\n",
    "          \"f1Score\": 0.4788087,\n",
    "          \"falsePositiveRate\": 0.02764188,\n",
    "          \"truePositiveCount\": \"209\",\n",
    "          \"falsePositiveCount\": \"113\",\n",
    "          \"falseNegativeCount\": \"342\",\n",
    "          \"trueNegativeCount\": \"3975\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5685674,\n",
    "          \"recall\": 0.36842105,\n",
    "          \"precision\": 0.650641,\n",
    "          \"f1Score\": 0.47045192,\n",
    "          \"falsePositiveRate\": 0.026663406,\n",
    "          \"truePositiveCount\": \"203\",\n",
    "          \"falsePositiveCount\": \"109\",\n",
    "          \"falseNegativeCount\": \"348\",\n",
    "          \"trueNegativeCount\": \"3979\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5732897,\n",
    "          \"recall\": 0.35934663,\n",
    "          \"precision\": 0.6578073,\n",
    "          \"f1Score\": 0.46478873,\n",
    "          \"falsePositiveRate\": 0.025195695,\n",
    "          \"truePositiveCount\": \"198\",\n",
    "          \"falsePositiveCount\": \"103\",\n",
    "          \"falseNegativeCount\": \"353\",\n",
    "          \"trueNegativeCount\": \"3985\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5798386,\n",
    "          \"recall\": 0.34845734,\n",
    "          \"precision\": 0.65753424,\n",
    "          \"f1Score\": 0.455516,\n",
    "          \"falsePositiveRate\": 0.02446184,\n",
    "          \"truePositiveCount\": \"192\",\n",
    "          \"falsePositiveCount\": \"100\",\n",
    "          \"falseNegativeCount\": \"359\",\n",
    "          \"trueNegativeCount\": \"3988\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5841479,\n",
    "          \"recall\": 0.33938295,\n",
    "          \"precision\": 0.6561403,\n",
    "          \"f1Score\": 0.4473684,\n",
    "          \"falsePositiveRate\": 0.023972603,\n",
    "          \"truePositiveCount\": \"187\",\n",
    "          \"falsePositiveCount\": \"98\",\n",
    "          \"falseNegativeCount\": \"364\",\n",
    "          \"trueNegativeCount\": \"3990\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.58893955,\n",
    "          \"recall\": 0.32849365,\n",
    "          \"precision\": 0.6630037,\n",
    "          \"f1Score\": 0.4393204,\n",
    "          \"falsePositiveRate\": 0.022504892,\n",
    "          \"truePositiveCount\": \"181\",\n",
    "          \"falsePositiveCount\": \"92\",\n",
    "          \"falseNegativeCount\": \"370\",\n",
    "          \"trueNegativeCount\": \"3996\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.59818196,\n",
    "          \"recall\": 0.31941923,\n",
    "          \"precision\": 0.6743295,\n",
    "          \"f1Score\": 0.43349755,\n",
    "          \"falsePositiveRate\": 0.020792564,\n",
    "          \"truePositiveCount\": \"176\",\n",
    "          \"falsePositiveCount\": \"85\",\n",
    "          \"falseNegativeCount\": \"375\",\n",
    "          \"trueNegativeCount\": \"4003\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.60704446,\n",
    "          \"recall\": 0.30852994,\n",
    "          \"precision\": 0.6854839,\n",
    "          \"f1Score\": 0.42553192,\n",
    "          \"falsePositiveRate\": 0.019080235,\n",
    "          \"truePositiveCount\": \"170\",\n",
    "          \"falsePositiveCount\": \"78\",\n",
    "          \"falseNegativeCount\": \"381\",\n",
    "          \"trueNegativeCount\": \"4010\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6096991,\n",
    "          \"recall\": 0.29945552,\n",
    "          \"precision\": 0.6818182,\n",
    "          \"f1Score\": 0.41614124,\n",
    "          \"falsePositiveRate\": 0.018835617,\n",
    "          \"truePositiveCount\": \"165\",\n",
    "          \"falsePositiveCount\": \"77\",\n",
    "          \"falseNegativeCount\": \"386\",\n",
    "          \"trueNegativeCount\": \"4011\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6194289,\n",
    "          \"recall\": 0.28856623,\n",
    "          \"precision\": 0.6913043,\n",
    "          \"f1Score\": 0.4071703,\n",
    "          \"falsePositiveRate\": 0.017367907,\n",
    "          \"truePositiveCount\": \"159\",\n",
    "          \"falsePositiveCount\": \"71\",\n",
    "          \"falseNegativeCount\": \"392\",\n",
    "          \"trueNegativeCount\": \"4017\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6299003,\n",
    "          \"recall\": 0.27949184,\n",
    "          \"precision\": 0.6936937,\n",
    "          \"f1Score\": 0.3984476,\n",
    "          \"falsePositiveRate\": 0.01663405,\n",
    "          \"truePositiveCount\": \"154\",\n",
    "          \"falsePositiveCount\": \"68\",\n",
    "          \"falseNegativeCount\": \"397\",\n",
    "          \"trueNegativeCount\": \"4020\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6326805,\n",
    "          \"recall\": 0.26860255,\n",
    "          \"precision\": 0.6883721,\n",
    "          \"f1Score\": 0.38642296,\n",
    "          \"falsePositiveRate\": 0.016389433,\n",
    "          \"truePositiveCount\": \"148\",\n",
    "          \"falsePositiveCount\": \"67\",\n",
    "          \"falseNegativeCount\": \"403\",\n",
    "          \"trueNegativeCount\": \"4021\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6446329,\n",
    "          \"recall\": 0.25952813,\n",
    "          \"precision\": 0.71859294,\n",
    "          \"f1Score\": 0.38133332,\n",
    "          \"falsePositiveRate\": 0.01369863,\n",
    "          \"truePositiveCount\": \"143\",\n",
    "          \"falsePositiveCount\": \"56\",\n",
    "          \"falseNegativeCount\": \"408\",\n",
    "          \"trueNegativeCount\": \"4032\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6525863,\n",
    "          \"recall\": 0.24863884,\n",
    "          \"precision\": 0.7287234,\n",
    "          \"f1Score\": 0.37077132,\n",
    "          \"falsePositiveRate\": 0.012475538,\n",
    "          \"truePositiveCount\": \"137\",\n",
    "          \"falsePositiveCount\": \"51\",\n",
    "          \"falseNegativeCount\": \"414\",\n",
    "          \"trueNegativeCount\": \"4037\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6613302,\n",
    "          \"recall\": 0.23956443,\n",
    "          \"precision\": 0.73333335,\n",
    "          \"f1Score\": 0.3611491,\n",
    "          \"falsePositiveRate\": 0.011741683,\n",
    "          \"truePositiveCount\": \"132\",\n",
    "          \"falsePositiveCount\": \"48\",\n",
    "          \"falseNegativeCount\": \"419\",\n",
    "          \"trueNegativeCount\": \"4040\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6681088,\n",
    "          \"recall\": 0.22867514,\n",
    "          \"precision\": 0.73255813,\n",
    "          \"f1Score\": 0.34854773,\n",
    "          \"falsePositiveRate\": 0.011252446,\n",
    "          \"truePositiveCount\": \"126\",\n",
    "          \"falsePositiveCount\": \"46\",\n",
    "          \"falseNegativeCount\": \"425\",\n",
    "          \"trueNegativeCount\": \"4042\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6787454,\n",
    "          \"recall\": 0.21960072,\n",
    "          \"precision\": 0.7378049,\n",
    "          \"f1Score\": 0.33846155,\n",
    "          \"falsePositiveRate\": 0.010518591,\n",
    "          \"truePositiveCount\": \"121\",\n",
    "          \"falsePositiveCount\": \"43\",\n",
    "          \"falseNegativeCount\": \"430\",\n",
    "          \"trueNegativeCount\": \"4045\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.68742466,\n",
    "          \"recall\": 0.20871143,\n",
    "          \"precision\": 0.73717946,\n",
    "          \"f1Score\": 0.32531825,\n",
    "          \"falsePositiveRate\": 0.010029354,\n",
    "          \"truePositiveCount\": \"115\",\n",
    "          \"falsePositiveCount\": \"41\",\n",
    "          \"falseNegativeCount\": \"436\",\n",
    "          \"trueNegativeCount\": \"4047\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.69688004,\n",
    "          \"recall\": 0.19963703,\n",
    "          \"precision\": 0.7482993,\n",
    "          \"f1Score\": 0.31518623,\n",
    "          \"falsePositiveRate\": 0.009050881,\n",
    "          \"truePositiveCount\": \"110\",\n",
    "          \"falsePositiveCount\": \"37\",\n",
    "          \"falseNegativeCount\": \"441\",\n",
    "          \"trueNegativeCount\": \"4051\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.70484525,\n",
    "          \"recall\": 0.18874773,\n",
    "          \"precision\": 0.7647059,\n",
    "          \"f1Score\": 0.30276564,\n",
    "          \"falsePositiveRate\": 0.007827789,\n",
    "          \"truePositiveCount\": \"104\",\n",
    "          \"falsePositiveCount\": \"32\",\n",
    "          \"falseNegativeCount\": \"447\",\n",
    "          \"trueNegativeCount\": \"4056\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.71349317,\n",
    "          \"recall\": 0.17967331,\n",
    "          \"precision\": 0.76153845,\n",
    "          \"f1Score\": 0.2907489,\n",
    "          \"falsePositiveRate\": 0.00758317,\n",
    "          \"truePositiveCount\": \"99\",\n",
    "          \"falsePositiveCount\": \"31\",\n",
    "          \"falseNegativeCount\": \"452\",\n",
    "          \"trueNegativeCount\": \"4057\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.72145003,\n",
    "          \"recall\": 0.16878402,\n",
    "          \"precision\": 0.75609756,\n",
    "          \"f1Score\": 0.27596438,\n",
    "          \"falsePositiveRate\": 0.007338552,\n",
    "          \"truePositiveCount\": \"93\",\n",
    "          \"falsePositiveCount\": \"30\",\n",
    "          \"falseNegativeCount\": \"458\",\n",
    "          \"trueNegativeCount\": \"4058\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.72871804,\n",
    "          \"recall\": 0.15970962,\n",
    "          \"precision\": 0.76521736,\n",
    "          \"f1Score\": 0.26426426,\n",
    "          \"falsePositiveRate\": 0.0066046966,\n",
    "          \"truePositiveCount\": \"88\",\n",
    "          \"falsePositiveCount\": \"27\",\n",
    "          \"falseNegativeCount\": \"463\",\n",
    "          \"trueNegativeCount\": \"4061\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.74161375,\n",
    "          \"recall\": 0.14882033,\n",
    "          \"precision\": 0.76635516,\n",
    "          \"f1Score\": 0.24924012,\n",
    "          \"falsePositiveRate\": 0.00611546,\n",
    "          \"truePositiveCount\": \"82\",\n",
    "          \"falsePositiveCount\": \"25\",\n",
    "          \"falseNegativeCount\": \"469\",\n",
    "          \"trueNegativeCount\": \"4063\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.74766874,\n",
    "          \"recall\": 0.13974592,\n",
    "          \"precision\": 0.78571427,\n",
    "          \"f1Score\": 0.23728813,\n",
    "          \"falsePositiveRate\": 0.0051369863,\n",
    "          \"truePositiveCount\": \"77\",\n",
    "          \"falsePositiveCount\": \"21\",\n",
    "          \"falseNegativeCount\": \"474\",\n",
    "          \"trueNegativeCount\": \"4067\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.75845456,\n",
    "          \"recall\": 0.12885663,\n",
    "          \"precision\": 0.8068182,\n",
    "          \"f1Score\": 0.22222222,\n",
    "          \"falsePositiveRate\": 0.0041585127,\n",
    "          \"truePositiveCount\": \"71\",\n",
    "          \"falsePositiveCount\": \"17\",\n",
    "          \"falseNegativeCount\": \"480\",\n",
    "          \"trueNegativeCount\": \"4071\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7666716,\n",
    "          \"recall\": 0.11978222,\n",
    "          \"precision\": 0.80487806,\n",
    "          \"f1Score\": 0.2085308,\n",
    "          \"falsePositiveRate\": 0.0039138943,\n",
    "          \"truePositiveCount\": \"66\",\n",
    "          \"falsePositiveCount\": \"16\",\n",
    "          \"falseNegativeCount\": \"485\",\n",
    "          \"trueNegativeCount\": \"4072\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7772814,\n",
    "          \"recall\": 0.108892925,\n",
    "          \"precision\": 0.7894737,\n",
    "          \"f1Score\": 0.19138756,\n",
    "          \"falsePositiveRate\": 0.0039138943,\n",
    "          \"truePositiveCount\": \"60\",\n",
    "          \"falsePositiveCount\": \"16\",\n",
    "          \"falseNegativeCount\": \"491\",\n",
    "          \"trueNegativeCount\": \"4072\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.78661436,\n",
    "          \"recall\": 0.09981851,\n",
    "          \"precision\": 0.8088235,\n",
    "          \"f1Score\": 0.17770597,\n",
    "          \"falsePositiveRate\": 0.003180039,\n",
    "          \"truePositiveCount\": \"55\",\n",
    "          \"falsePositiveCount\": \"13\",\n",
    "          \"falseNegativeCount\": \"496\",\n",
    "          \"trueNegativeCount\": \"4075\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8050728,\n",
    "          \"recall\": 0.08892922,\n",
    "          \"precision\": 0.8448276,\n",
    "          \"f1Score\": 0.16091955,\n",
    "          \"falsePositiveRate\": 0.0022015655,\n",
    "          \"truePositiveCount\": \"49\",\n",
    "          \"falsePositiveCount\": \"9\",\n",
    "          \"falseNegativeCount\": \"502\",\n",
    "          \"trueNegativeCount\": \"4079\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8127946,\n",
    "          \"recall\": 0.07985481,\n",
    "          \"precision\": 0.8301887,\n",
    "          \"f1Score\": 0.14569536,\n",
    "          \"falsePositiveRate\": 0.0022015655,\n",
    "          \"truePositiveCount\": \"44\",\n",
    "          \"falsePositiveCount\": \"9\",\n",
    "          \"falseNegativeCount\": \"507\",\n",
    "          \"trueNegativeCount\": \"4079\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.82794523,\n",
    "          \"recall\": 0.06896552,\n",
    "          \"precision\": 0.82608694,\n",
    "          \"f1Score\": 0.12730318,\n",
    "          \"falsePositiveRate\": 0.0019569471,\n",
    "          \"truePositiveCount\": \"38\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"513\",\n",
    "          \"trueNegativeCount\": \"4080\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.83858025,\n",
    "          \"recall\": 0.05989111,\n",
    "          \"precision\": 0.84615386,\n",
    "          \"f1Score\": 0.11186441,\n",
    "          \"falsePositiveRate\": 0.0014677104,\n",
    "          \"truePositiveCount\": \"33\",\n",
    "          \"falsePositiveCount\": \"6\",\n",
    "          \"falseNegativeCount\": \"518\",\n",
    "          \"trueNegativeCount\": \"4082\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.85226774,\n",
    "          \"recall\": 0.049001817,\n",
    "          \"precision\": 0.87096775,\n",
    "          \"f1Score\": 0.0927835,\n",
    "          \"falsePositiveRate\": 0.0009784736,\n",
    "          \"truePositiveCount\": \"27\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"524\",\n",
    "          \"trueNegativeCount\": \"4084\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.86955357,\n",
    "          \"recall\": 0.039927404,\n",
    "          \"precision\": 0.88,\n",
    "          \"f1Score\": 0.07638889,\n",
    "          \"falsePositiveRate\": 0.0007338552,\n",
    "          \"truePositiveCount\": \"22\",\n",
    "          \"falsePositiveCount\": \"3\",\n",
    "          \"falseNegativeCount\": \"529\",\n",
    "          \"trueNegativeCount\": \"4085\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8754055,\n",
    "          \"recall\": 0.029038113,\n",
    "          \"precision\": 0.84210527,\n",
    "          \"f1Score\": 0.056140352,\n",
    "          \"falsePositiveRate\": 0.0007338552,\n",
    "          \"truePositiveCount\": \"16\",\n",
    "          \"falsePositiveCount\": \"3\",\n",
    "          \"falseNegativeCount\": \"535\",\n",
    "          \"trueNegativeCount\": \"4085\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.888999,\n",
    "          \"recall\": 0.019963702,\n",
    "          \"precision\": 0.78571427,\n",
    "          \"f1Score\": 0.038938053,\n",
    "          \"falsePositiveRate\": 0.0007338552,\n",
    "          \"truePositiveCount\": \"11\",\n",
    "          \"falsePositiveCount\": \"3\",\n",
    "          \"falseNegativeCount\": \"540\",\n",
    "          \"trueNegativeCount\": \"4085\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.912919,\n",
    "          \"recall\": 0.00907441,\n",
    "          \"precision\": 0.71428573,\n",
    "          \"f1Score\": 0.017921148,\n",
    "          \"falsePositiveRate\": 0.0004892368,\n",
    "          \"truePositiveCount\": \"5\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"546\",\n",
    "          \"trueNegativeCount\": \"4086\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9486238,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"551\",\n",
    "          \"trueNegativeCount\": \"4088\",\n",
    "          \"positionThreshold\": 1\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 1.0,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"551\",\n",
    "          \"trueNegativeCount\": \"4088\",\n",
    "          \"positionThreshold\": 1\n",
    "        }\n",
    "      ],\n",
    "      \"auRoc\": 0.71778876,\n",
    "      \"logLoss\": 0.993795\n",
    "    },\n",
    "    \"displayName\": \"2\"\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/7171857336139058652\",\n",
    "    \"annotationSpecId\": \"not available\",\n",
    "    \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "    \"evaluatedExampleCount\": 4639,\n",
    "    \"classificationEvaluationMetrics\": {\n",
    "      \"auPrc\": 0.97598535,\n",
    "      \"confidenceMetricsEntry\": [\n",
    "        {\n",
    "          \"recall\": 1.0,\n",
    "          \"precision\": 0.5,\n",
    "          \"f1Score\": 0.6666667,\n",
    "          \"falsePositiveRate\": 1.0,\n",
    "          \"truePositiveCount\": \"4639\",\n",
    "          \"falsePositiveCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.009267787,\n",
    "          \"recall\": 0.9997844,\n",
    "          \"precision\": 0.6205512,\n",
    "          \"f1Score\": 0.7657888,\n",
    "          \"falsePositiveRate\": 0.6113387,\n",
    "          \"truePositiveCount\": \"4638\",\n",
    "          \"falsePositiveCount\": \"2836\",\n",
    "          \"falseNegativeCount\": \"1\",\n",
    "          \"trueNegativeCount\": \"1803\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.08771089,\n",
    "          \"recall\": 0.99029964,\n",
    "          \"precision\": 0.77640694,\n",
    "          \"f1Score\": 0.87040544,\n",
    "          \"falsePositiveRate\": 0.28519076,\n",
    "          \"truePositiveCount\": \"4594\",\n",
    "          \"falsePositiveCount\": \"1323\",\n",
    "          \"falseNegativeCount\": \"45\",\n",
    "          \"trueNegativeCount\": \"3316\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.18867286,\n",
    "          \"recall\": 0.98016816,\n",
    "          \"precision\": 0.82268864,\n",
    "          \"f1Score\": 0.89455044,\n",
    "          \"falsePositiveRate\": 0.21125242,\n",
    "          \"truePositiveCount\": \"4547\",\n",
    "          \"falsePositiveCount\": \"980\",\n",
    "          \"falseNegativeCount\": \"92\",\n",
    "          \"trueNegativeCount\": \"3659\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.24711765,\n",
    "          \"recall\": 0.9702522,\n",
    "          \"precision\": 0.841623,\n",
    "          \"f1Score\": 0.9013718,\n",
    "          \"falsePositiveRate\": 0.18258245,\n",
    "          \"truePositiveCount\": \"4501\",\n",
    "          \"falsePositiveCount\": \"847\",\n",
    "          \"falseNegativeCount\": \"138\",\n",
    "          \"trueNegativeCount\": \"3792\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.29667678,\n",
    "          \"recall\": 0.96012074,\n",
    "          \"precision\": 0.85358375,\n",
    "          \"f1Score\": 0.90372324,\n",
    "          \"falsePositiveRate\": 0.16469067,\n",
    "          \"truePositiveCount\": \"4454\",\n",
    "          \"falsePositiveCount\": \"764\",\n",
    "          \"falseNegativeCount\": \"185\",\n",
    "          \"trueNegativeCount\": \"3875\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.33910435,\n",
    "          \"recall\": 0.9499892,\n",
    "          \"precision\": 0.8619206,\n",
    "          \"f1Score\": 0.9038146,\n",
    "          \"falsePositiveRate\": 0.15218797,\n",
    "          \"truePositiveCount\": \"4407\",\n",
    "          \"falsePositiveCount\": \"706\",\n",
    "          \"falseNegativeCount\": \"232\",\n",
    "          \"trueNegativeCount\": \"3933\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.3722213,\n",
    "          \"recall\": 0.94028884,\n",
    "          \"precision\": 0.87031126,\n",
    "          \"f1Score\": 0.9039478,\n",
    "          \"falsePositiveRate\": 0.14011641,\n",
    "          \"truePositiveCount\": \"4362\",\n",
    "          \"falsePositiveCount\": \"650\",\n",
    "          \"falseNegativeCount\": \"277\",\n",
    "          \"trueNegativeCount\": \"3989\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.41141066,\n",
    "          \"recall\": 0.93015736,\n",
    "          \"precision\": 0.87971455,\n",
    "          \"f1Score\": 0.90423304,\n",
    "          \"falsePositiveRate\": 0.12718259,\n",
    "          \"truePositiveCount\": \"4315\",\n",
    "          \"falsePositiveCount\": \"590\",\n",
    "          \"falseNegativeCount\": \"324\",\n",
    "          \"trueNegativeCount\": \"4049\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.44597256,\n",
    "          \"recall\": 0.920457,\n",
    "          \"precision\": 0.88921285,\n",
    "          \"f1Score\": 0.9045652,\n",
    "          \"falsePositiveRate\": 0.11467989,\n",
    "          \"truePositiveCount\": \"4270\",\n",
    "          \"falsePositiveCount\": \"532\",\n",
    "          \"falseNegativeCount\": \"369\",\n",
    "          \"trueNegativeCount\": \"4107\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.48088387,\n",
    "          \"recall\": 0.91010994,\n",
    "          \"precision\": 0.89829785,\n",
    "          \"f1Score\": 0.9041653,\n",
    "          \"falsePositiveRate\": 0.10303945,\n",
    "          \"truePositiveCount\": \"4222\",\n",
    "          \"falsePositiveCount\": \"478\",\n",
    "          \"falseNegativeCount\": \"417\",\n",
    "          \"trueNegativeCount\": \"4161\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5,\n",
    "          \"recall\": 0.90515196,\n",
    "          \"precision\": 0.90515196,\n",
    "          \"f1Score\": 0.90515196,\n",
    "          \"falsePositiveRate\": 0.09484803,\n",
    "          \"truePositiveCount\": \"4199\",\n",
    "          \"falsePositiveCount\": \"440\",\n",
    "          \"falseNegativeCount\": \"440\",\n",
    "          \"trueNegativeCount\": \"4199\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.51082355,\n",
    "          \"recall\": 0.9006251,\n",
    "          \"precision\": 0.90688086,\n",
    "          \"f1Score\": 0.90374213,\n",
    "          \"falsePositiveRate\": 0.09247683,\n",
    "          \"truePositiveCount\": \"4178\",\n",
    "          \"falsePositiveCount\": \"429\",\n",
    "          \"falseNegativeCount\": \"461\",\n",
    "          \"trueNegativeCount\": \"4210\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.5395309,\n",
    "          \"recall\": 0.8902781,\n",
    "          \"precision\": 0.9145261,\n",
    "          \"f1Score\": 0.9022392,\n",
    "          \"falsePositiveRate\": 0.083207585,\n",
    "          \"truePositiveCount\": \"4130\",\n",
    "          \"falsePositiveCount\": \"386\",\n",
    "          \"falseNegativeCount\": \"509\",\n",
    "          \"trueNegativeCount\": \"4253\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.569127,\n",
    "          \"recall\": 0.88014656,\n",
    "          \"precision\": 0.92083895,\n",
    "          \"f1Score\": 0.90003306,\n",
    "          \"falsePositiveRate\": 0.07566286,\n",
    "          \"truePositiveCount\": \"4083\",\n",
    "          \"falsePositiveCount\": \"351\",\n",
    "          \"falseNegativeCount\": \"556\",\n",
    "          \"trueNegativeCount\": \"4288\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.59819055,\n",
    "          \"recall\": 0.8700151,\n",
    "          \"precision\": 0.9284564,\n",
    "          \"f1Score\": 0.8982862,\n",
    "          \"falsePositiveRate\": 0.06704031,\n",
    "          \"truePositiveCount\": \"4036\",\n",
    "          \"falsePositiveCount\": \"311\",\n",
    "          \"falseNegativeCount\": \"603\",\n",
    "          \"trueNegativeCount\": \"4328\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6267493,\n",
    "          \"recall\": 0.86009914,\n",
    "          \"precision\": 0.9350832,\n",
    "          \"f1Score\": 0.8960252,\n",
    "          \"falsePositiveRate\": 0.059711143,\n",
    "          \"truePositiveCount\": \"3990\",\n",
    "          \"falsePositiveCount\": \"277\",\n",
    "          \"falseNegativeCount\": \"649\",\n",
    "          \"trueNegativeCount\": \"4362\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.6483357,\n",
    "          \"recall\": 0.84996766,\n",
    "          \"precision\": 0.9401526,\n",
    "          \"f1Score\": 0.8927884,\n",
    "          \"falsePositiveRate\": 0.05410649,\n",
    "          \"truePositiveCount\": \"3943\",\n",
    "          \"falsePositiveCount\": \"251\",\n",
    "          \"falseNegativeCount\": \"696\",\n",
    "          \"trueNegativeCount\": \"4388\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.68742466,\n",
    "          \"recall\": 0.8404829,\n",
    "          \"precision\": 0.95051193,\n",
    "          \"f1Score\": 0.8921176,\n",
    "          \"falsePositiveRate\": 0.04375943,\n",
    "          \"truePositiveCount\": \"3899\",\n",
    "          \"falsePositiveCount\": \"203\",\n",
    "          \"falseNegativeCount\": \"740\",\n",
    "          \"trueNegativeCount\": \"4436\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7178758,\n",
    "          \"recall\": 0.83035135,\n",
    "          \"precision\": 0.95630586,\n",
    "          \"f1Score\": 0.8888889,\n",
    "          \"falsePositiveRate\": 0.03793921,\n",
    "          \"truePositiveCount\": \"3852\",\n",
    "          \"falsePositiveCount\": \"176\",\n",
    "          \"falseNegativeCount\": \"787\",\n",
    "          \"trueNegativeCount\": \"4463\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7454512,\n",
    "          \"recall\": 0.8200043,\n",
    "          \"precision\": 0.9620637,\n",
    "          \"f1Score\": 0.8853718,\n",
    "          \"falsePositiveRate\": 0.032334555,\n",
    "          \"truePositiveCount\": \"3804\",\n",
    "          \"falsePositiveCount\": \"150\",\n",
    "          \"falseNegativeCount\": \"835\",\n",
    "          \"trueNegativeCount\": \"4489\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.7702336,\n",
    "          \"recall\": 0.8105195,\n",
    "          \"precision\": 0.9685729,\n",
    "          \"f1Score\": 0.8825255,\n",
    "          \"falsePositiveRate\": 0.02629877,\n",
    "          \"truePositiveCount\": \"3760\",\n",
    "          \"falsePositiveCount\": \"122\",\n",
    "          \"falseNegativeCount\": \"879\",\n",
    "          \"trueNegativeCount\": \"4517\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.79181457,\n",
    "          \"recall\": 0.80038804,\n",
    "          \"precision\": 0.972244,\n",
    "          \"f1Score\": 0.87798536,\n",
    "          \"falsePositiveRate\": 0.022849752,\n",
    "          \"truePositiveCount\": \"3713\",\n",
    "          \"falsePositiveCount\": \"106\",\n",
    "          \"falseNegativeCount\": \"926\",\n",
    "          \"trueNegativeCount\": \"4533\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.80905926,\n",
    "          \"recall\": 0.7902565,\n",
    "          \"precision\": 0.9755189,\n",
    "          \"f1Score\": 0.873169,\n",
    "          \"falsePositiveRate\": 0.01983186,\n",
    "          \"truePositiveCount\": \"3666\",\n",
    "          \"falsePositiveCount\": \"92\",\n",
    "          \"falseNegativeCount\": \"973\",\n",
    "          \"trueNegativeCount\": \"4547\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.82475233,\n",
    "          \"recall\": 0.7803406,\n",
    "          \"precision\": 0.97574127,\n",
    "          \"f1Score\": 0.86716974,\n",
    "          \"falsePositiveRate\": 0.019400733,\n",
    "          \"truePositiveCount\": \"3620\",\n",
    "          \"falsePositiveCount\": \"90\",\n",
    "          \"falseNegativeCount\": \"1019\",\n",
    "          \"trueNegativeCount\": \"4549\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.84347624,\n",
    "          \"recall\": 0.7702091,\n",
    "          \"precision\": 0.97863597,\n",
    "          \"f1Score\": 0.86200243,\n",
    "          \"falsePositiveRate\": 0.01681397,\n",
    "          \"truePositiveCount\": \"3573\",\n",
    "          \"falsePositiveCount\": \"78\",\n",
    "          \"falseNegativeCount\": \"1066\",\n",
    "          \"trueNegativeCount\": \"4561\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8612723,\n",
    "          \"recall\": 0.7602932,\n",
    "          \"precision\": 0.9799944,\n",
    "          \"f1Score\": 0.8562758,\n",
    "          \"falsePositiveRate\": 0.015520587,\n",
    "          \"truePositiveCount\": \"3527\",\n",
    "          \"falsePositiveCount\": \"72\",\n",
    "          \"falseNegativeCount\": \"1112\",\n",
    "          \"trueNegativeCount\": \"4567\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.87450475,\n",
    "          \"recall\": 0.75016165,\n",
    "          \"precision\": 0.9811108,\n",
    "          \"f1Score\": 0.8502321,\n",
    "          \"falsePositiveRate\": 0.014442768,\n",
    "          \"truePositiveCount\": \"3480\",\n",
    "          \"falsePositiveCount\": \"67\",\n",
    "          \"falseNegativeCount\": \"1159\",\n",
    "          \"trueNegativeCount\": \"4572\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.8876167,\n",
    "          \"recall\": 0.74003017,\n",
    "          \"precision\": 0.9825415,\n",
    "          \"f1Score\": 0.8442149,\n",
    "          \"falsePositiveRate\": 0.013149385,\n",
    "          \"truePositiveCount\": \"3433\",\n",
    "          \"falsePositiveCount\": \"61\",\n",
    "          \"falseNegativeCount\": \"1206\",\n",
    "          \"trueNegativeCount\": \"4578\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.89729506,\n",
    "          \"recall\": 0.7303298,\n",
    "          \"precision\": 0.98402554,\n",
    "          \"f1Score\": 0.8384063,\n",
    "          \"falsePositiveRate\": 0.011856004,\n",
    "          \"truePositiveCount\": \"3388\",\n",
    "          \"falsePositiveCount\": \"55\",\n",
    "          \"falseNegativeCount\": \"1251\",\n",
    "          \"trueNegativeCount\": \"4584\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9069939,\n",
    "          \"recall\": 0.72041386,\n",
    "          \"precision\": 0.9858407,\n",
    "          \"f1Score\": 0.8324823,\n",
    "          \"falsePositiveRate\": 0.010347057,\n",
    "          \"truePositiveCount\": \"3342\",\n",
    "          \"falsePositiveCount\": \"48\",\n",
    "          \"falseNegativeCount\": \"1297\",\n",
    "          \"trueNegativeCount\": \"4591\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.91544837,\n",
    "          \"recall\": 0.7102824,\n",
    "          \"precision\": 0.987118,\n",
    "          \"f1Score\": 0.8261251,\n",
    "          \"falsePositiveRate\": 0.009269239,\n",
    "          \"truePositiveCount\": \"3295\",\n",
    "          \"falsePositiveCount\": \"43\",\n",
    "          \"falseNegativeCount\": \"1344\",\n",
    "          \"trueNegativeCount\": \"4596\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9246296,\n",
    "          \"recall\": 0.700582,\n",
    "          \"precision\": 0.9896468,\n",
    "          \"f1Score\": 0.8203963,\n",
    "          \"falsePositiveRate\": 0.007329166,\n",
    "          \"truePositiveCount\": \"3250\",\n",
    "          \"falsePositiveCount\": \"34\",\n",
    "          \"falseNegativeCount\": \"1389\",\n",
    "          \"trueNegativeCount\": \"4605\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.932719,\n",
    "          \"recall\": 0.6900194,\n",
    "          \"precision\": 0.99071497,\n",
    "          \"f1Score\": 0.8134689,\n",
    "          \"falsePositiveRate\": 0.006466911,\n",
    "          \"truePositiveCount\": \"3201\",\n",
    "          \"falsePositiveCount\": \"30\",\n",
    "          \"falseNegativeCount\": \"1438\",\n",
    "          \"trueNegativeCount\": \"4609\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.93940556,\n",
    "          \"recall\": 0.6807502,\n",
    "          \"precision\": 0.9921458,\n",
    "          \"f1Score\": 0.80746615,\n",
    "          \"falsePositiveRate\": 0.0053890925,\n",
    "          \"truePositiveCount\": \"3158\",\n",
    "          \"falsePositiveCount\": \"25\",\n",
    "          \"falseNegativeCount\": \"1481\",\n",
    "          \"trueNegativeCount\": \"4614\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9436984,\n",
    "          \"recall\": 0.6704031,\n",
    "          \"precision\": 0.99234205,\n",
    "          \"f1Score\": 0.8002058,\n",
    "          \"falsePositiveRate\": 0.0051735286,\n",
    "          \"truePositiveCount\": \"3110\",\n",
    "          \"falsePositiveCount\": \"24\",\n",
    "          \"falseNegativeCount\": \"1529\",\n",
    "          \"trueNegativeCount\": \"4615\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9486017,\n",
    "          \"recall\": 0.6602716,\n",
    "          \"precision\": 0.992547,\n",
    "          \"f1Score\": 0.7930097,\n",
    "          \"falsePositiveRate\": 0.004957965,\n",
    "          \"truePositiveCount\": \"3063\",\n",
    "          \"falsePositiveCount\": \"23\",\n",
    "          \"falseNegativeCount\": \"1576\",\n",
    "          \"trueNegativeCount\": \"4616\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95302504,\n",
    "          \"recall\": 0.6501401,\n",
    "          \"precision\": 0.9937397,\n",
    "          \"f1Score\": 0.78603077,\n",
    "          \"falsePositiveRate\": 0.0040957103,\n",
    "          \"truePositiveCount\": \"3016\",\n",
    "          \"falsePositiveCount\": \"19\",\n",
    "          \"falseNegativeCount\": \"1623\",\n",
    "          \"trueNegativeCount\": \"4620\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95584136,\n",
    "          \"recall\": 0.6400086,\n",
    "          \"precision\": 0.99530673,\n",
    "          \"f1Score\": 0.7790606,\n",
    "          \"falsePositiveRate\": 0.003017892,\n",
    "          \"truePositiveCount\": \"2969\",\n",
    "          \"falsePositiveCount\": \"14\",\n",
    "          \"falseNegativeCount\": \"1670\",\n",
    "          \"trueNegativeCount\": \"4625\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.95881057,\n",
    "          \"recall\": 0.6300927,\n",
    "          \"precision\": 0.9955722,\n",
    "          \"f1Score\": 0.7717492,\n",
    "          \"falsePositiveRate\": 0.002802328,\n",
    "          \"truePositiveCount\": \"2923\",\n",
    "          \"falsePositiveCount\": \"13\",\n",
    "          \"falseNegativeCount\": \"1716\",\n",
    "          \"trueNegativeCount\": \"4626\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96166563,\n",
    "          \"recall\": 0.62082344,\n",
    "          \"precision\": 0.9961951,\n",
    "          \"f1Score\": 0.76494026,\n",
    "          \"falsePositiveRate\": 0.0023712006,\n",
    "          \"truePositiveCount\": \"2880\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1759\",\n",
    "          \"trueNegativeCount\": \"4628\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96420497,\n",
    "          \"recall\": 0.61026084,\n",
    "          \"precision\": 0.9961295,\n",
    "          \"f1Score\": 0.75685066,\n",
    "          \"falsePositiveRate\": 0.0023712006,\n",
    "          \"truePositiveCount\": \"2831\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1808\",\n",
    "          \"trueNegativeCount\": \"4628\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96650803,\n",
    "          \"recall\": 0.5999138,\n",
    "          \"precision\": 0.996063,\n",
    "          \"f1Score\": 0.7488228,\n",
    "          \"falsePositiveRate\": 0.0023712006,\n",
    "          \"truePositiveCount\": \"2783\",\n",
    "          \"falsePositiveCount\": \"11\",\n",
    "          \"falseNegativeCount\": \"1856\",\n",
    "          \"trueNegativeCount\": \"4628\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.96897614,\n",
    "          \"recall\": 0.59042895,\n",
    "          \"precision\": 0.9963623,\n",
    "          \"f1Score\": 0.74147266,\n",
    "          \"falsePositiveRate\": 0.002155637,\n",
    "          \"truePositiveCount\": \"2739\",\n",
    "          \"falsePositiveCount\": \"10\",\n",
    "          \"falseNegativeCount\": \"1900\",\n",
    "          \"trueNegativeCount\": \"4629\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97092766,\n",
    "          \"recall\": 0.5807286,\n",
    "          \"precision\": 0.99667037,\n",
    "          \"f1Score\": 0.73385996,\n",
    "          \"falsePositiveRate\": 0.0019400733,\n",
    "          \"truePositiveCount\": \"2694\",\n",
    "          \"falsePositiveCount\": \"9\",\n",
    "          \"falseNegativeCount\": \"1945\",\n",
    "          \"trueNegativeCount\": \"4630\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9727514,\n",
    "          \"recall\": 0.5699504,\n",
    "          \"precision\": 0.9969834,\n",
    "          \"f1Score\": 0.7252777,\n",
    "          \"falsePositiveRate\": 0.0017245096,\n",
    "          \"truePositiveCount\": \"2644\",\n",
    "          \"falsePositiveCount\": \"8\",\n",
    "          \"falseNegativeCount\": \"1995\",\n",
    "          \"trueNegativeCount\": \"4631\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9747563,\n",
    "          \"recall\": 0.56046563,\n",
    "          \"precision\": 0.99731493,\n",
    "          \"f1Score\": 0.7176373,\n",
    "          \"falsePositiveRate\": 0.001508946,\n",
    "          \"truePositiveCount\": \"2600\",\n",
    "          \"falsePositiveCount\": \"7\",\n",
    "          \"falseNegativeCount\": \"2039\",\n",
    "          \"trueNegativeCount\": \"4632\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9766838,\n",
    "          \"recall\": 0.549903,\n",
    "          \"precision\": 0.9976535,\n",
    "          \"f1Score\": 0.709005,\n",
    "          \"falsePositiveRate\": 0.0012933821,\n",
    "          \"truePositiveCount\": \"2551\",\n",
    "          \"falsePositiveCount\": \"6\",\n",
    "          \"falseNegativeCount\": \"2088\",\n",
    "          \"trueNegativeCount\": \"4633\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.97776866,\n",
    "          \"recall\": 0.54063374,\n",
    "          \"precision\": 0.99801034,\n",
    "          \"f1Score\": 0.7013423,\n",
    "          \"falsePositiveRate\": 0.0010778185,\n",
    "          \"truePositiveCount\": \"2508\",\n",
    "          \"falsePositiveCount\": \"5\",\n",
    "          \"falseNegativeCount\": \"2131\",\n",
    "          \"trueNegativeCount\": \"4634\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9792647,\n",
    "          \"recall\": 0.53050226,\n",
    "          \"precision\": 0.9979724,\n",
    "          \"f1Score\": 0.6927516,\n",
    "          \"falsePositiveRate\": 0.0010778185,\n",
    "          \"truePositiveCount\": \"2461\",\n",
    "          \"falsePositiveCount\": \"5\",\n",
    "          \"falseNegativeCount\": \"2178\",\n",
    "          \"trueNegativeCount\": \"4634\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9804138,\n",
    "          \"recall\": 0.5203708,\n",
    "          \"precision\": 0.99834573,\n",
    "          \"f1Score\": 0.6841434,\n",
    "          \"falsePositiveRate\": 0.0008622548,\n",
    "          \"truePositiveCount\": \"2414\",\n",
    "          \"falsePositiveCount\": \"4\",\n",
    "          \"falseNegativeCount\": \"2225\",\n",
    "          \"trueNegativeCount\": \"4635\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9814273,\n",
    "          \"recall\": 0.51045483,\n",
    "          \"precision\": 0.9991561,\n",
    "          \"f1Score\": 0.6757027,\n",
    "          \"falsePositiveRate\": 0.0004311274,\n",
    "          \"truePositiveCount\": \"2368\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"2271\",\n",
    "          \"trueNegativeCount\": \"4637\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98243207,\n",
    "          \"recall\": 0.5005389,\n",
    "          \"precision\": 0.9991394,\n",
    "          \"f1Score\": 0.6669539,\n",
    "          \"falsePositiveRate\": 0.0004311274,\n",
    "          \"truePositiveCount\": \"2322\",\n",
    "          \"falsePositiveCount\": \"2\",\n",
    "          \"falseNegativeCount\": \"2317\",\n",
    "          \"trueNegativeCount\": \"4637\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9835001,\n",
    "          \"recall\": 0.49019185,\n",
    "          \"precision\": 0.9995604,\n",
    "          \"f1Score\": 0.6577958,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2274\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2365\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9844133,\n",
    "          \"recall\": 0.4804915,\n",
    "          \"precision\": 0.9995516,\n",
    "          \"f1Score\": 0.6490028,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2229\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2410\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98528725,\n",
    "          \"recall\": 0.47079113,\n",
    "          \"precision\": 0.99954236,\n",
    "          \"f1Score\": 0.6400938,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2184\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2455\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.98630506,\n",
    "          \"recall\": 0.46001294,\n",
    "          \"precision\": 0.9995316,\n",
    "          \"f1Score\": 0.6300561,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2134\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2505\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9871925,\n",
    "          \"recall\": 0.44988143,\n",
    "          \"precision\": 0.9995211,\n",
    "          \"f1Score\": 0.6204846,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2087\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2552\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9877509,\n",
    "          \"recall\": 0.44082776,\n",
    "          \"precision\": 0.99951124,\n",
    "          \"f1Score\": 0.6118175,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"2045\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2594\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9885219,\n",
    "          \"recall\": 0.43048072,\n",
    "          \"precision\": 0.9994995,\n",
    "          \"f1Score\": 0.6017779,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1997\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2642\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9892192,\n",
    "          \"recall\": 0.4199181,\n",
    "          \"precision\": 0.9994869,\n",
    "          \"f1Score\": 0.5913783,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1948\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2691\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9897657,\n",
    "          \"recall\": 0.41000214,\n",
    "          \"precision\": 0.9994745,\n",
    "          \"f1Score\": 0.5814735,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1902\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2737\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9902156,\n",
    "          \"recall\": 0.40051734,\n",
    "          \"precision\": 0.99946207,\n",
    "          \"f1Score\": 0.57186824,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1858\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2781\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.990671,\n",
    "          \"recall\": 0.3901703,\n",
    "          \"precision\": 0.9994478,\n",
    "          \"f1Score\": 0.5612403,\n",
    "          \"falsePositiveRate\": 0.0002155637,\n",
    "          \"truePositiveCount\": \"1810\",\n",
    "          \"falsePositiveCount\": \"1\",\n",
    "          \"falseNegativeCount\": \"2829\",\n",
    "          \"trueNegativeCount\": \"4638\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99106044,\n",
    "          \"recall\": 0.38025436,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5509917,\n",
    "          \"truePositiveCount\": \"1764\",\n",
    "          \"falseNegativeCount\": \"2875\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99142385,\n",
    "          \"recall\": 0.36990732,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5400472,\n",
    "          \"truePositiveCount\": \"1716\",\n",
    "          \"falseNegativeCount\": \"2923\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.991821,\n",
    "          \"recall\": 0.35999137,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.52940243,\n",
    "          \"truePositiveCount\": \"1670\",\n",
    "          \"falseNegativeCount\": \"2969\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99219143,\n",
    "          \"recall\": 0.35007545,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.5186013,\n",
    "          \"truePositiveCount\": \"1624\",\n",
    "          \"falseNegativeCount\": \"3015\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99265563,\n",
    "          \"recall\": 0.3401595,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.50764036,\n",
    "          \"truePositiveCount\": \"1578\",\n",
    "          \"falseNegativeCount\": \"3061\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9930375,\n",
    "          \"recall\": 0.33045915,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.49675956,\n",
    "          \"truePositiveCount\": \"1533\",\n",
    "          \"falseNegativeCount\": \"3106\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9932301,\n",
    "          \"recall\": 0.32054323,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.48547176,\n",
    "          \"truePositiveCount\": \"1487\",\n",
    "          \"falseNegativeCount\": \"3152\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99349546,\n",
    "          \"recall\": 0.31041172,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.47376212,\n",
    "          \"truePositiveCount\": \"1440\",\n",
    "          \"falseNegativeCount\": \"3199\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.993766,\n",
    "          \"recall\": 0.30006468,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.461615,\n",
    "          \"truePositiveCount\": \"1392\",\n",
    "          \"falseNegativeCount\": \"3247\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9940096,\n",
    "          \"recall\": 0.29014874,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.44979113,\n",
    "          \"truePositiveCount\": \"1346\",\n",
    "          \"falseNegativeCount\": \"3293\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9942971,\n",
    "          \"recall\": 0.27980167,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.4372579,\n",
    "          \"truePositiveCount\": \"1298\",\n",
    "          \"falseNegativeCount\": \"3341\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9945464,\n",
    "          \"recall\": 0.26988575,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.42505518,\n",
    "          \"truePositiveCount\": \"1252\",\n",
    "          \"falseNegativeCount\": \"3387\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.994734,\n",
    "          \"recall\": 0.2601854,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.41293192,\n",
    "          \"truePositiveCount\": \"1207\",\n",
    "          \"falseNegativeCount\": \"3432\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9949861,\n",
    "          \"recall\": 0.25026944,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.40034482,\n",
    "          \"truePositiveCount\": \"1161\",\n",
    "          \"falseNegativeCount\": \"3478\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99523896,\n",
    "          \"recall\": 0.2399224,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.38699582,\n",
    "          \"truePositiveCount\": \"1113\",\n",
    "          \"falseNegativeCount\": \"3526\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99545944,\n",
    "          \"recall\": 0.23043759,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.37456203,\n",
    "          \"truePositiveCount\": \"1069\",\n",
    "          \"falseNegativeCount\": \"3570\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9956529,\n",
    "          \"recall\": 0.22009054,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.36077738,\n",
    "          \"truePositiveCount\": \"1021\",\n",
    "          \"falseNegativeCount\": \"3618\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9958653,\n",
    "          \"recall\": 0.21039017,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.34764025,\n",
    "          \"truePositiveCount\": \"976\",\n",
    "          \"falseNegativeCount\": \"3663\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99604857,\n",
    "          \"recall\": 0.20025867,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.33369252,\n",
    "          \"truePositiveCount\": \"929\",\n",
    "          \"falseNegativeCount\": \"3710\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99617386,\n",
    "          \"recall\": 0.19012718,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.31950733,\n",
    "          \"truePositiveCount\": \"882\",\n",
    "          \"falseNegativeCount\": \"3757\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9963188,\n",
    "          \"recall\": 0.18042682,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.3056976,\n",
    "          \"truePositiveCount\": \"837\",\n",
    "          \"falseNegativeCount\": \"3802\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99644965,\n",
    "          \"recall\": 0.17029533,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.29102966,\n",
    "          \"truePositiveCount\": \"790\",\n",
    "          \"falseNegativeCount\": \"3849\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99657893,\n",
    "          \"recall\": 0.16059496,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.27674592,\n",
    "          \"truePositiveCount\": \"745\",\n",
    "          \"falseNegativeCount\": \"3894\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99672806,\n",
    "          \"recall\": 0.1502479,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.2612444,\n",
    "          \"truePositiveCount\": \"697\",\n",
    "          \"falseNegativeCount\": \"3942\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9968369,\n",
    "          \"recall\": 0.14033197,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.24612476,\n",
    "          \"truePositiveCount\": \"651\",\n",
    "          \"falseNegativeCount\": \"3988\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9969655,\n",
    "          \"recall\": 0.13041604,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.23073989,\n",
    "          \"truePositiveCount\": \"605\",\n",
    "          \"falseNegativeCount\": \"4034\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9970963,\n",
    "          \"recall\": 0.12028454,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.21473928,\n",
    "          \"truePositiveCount\": \"558\",\n",
    "          \"falseNegativeCount\": \"4081\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9972023,\n",
    "          \"recall\": 0.110799745,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.19949543,\n",
    "          \"truePositiveCount\": \"514\",\n",
    "          \"falseNegativeCount\": \"4125\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9973033,\n",
    "          \"recall\": 0.100668244,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.18292205,\n",
    "          \"truePositiveCount\": \"467\",\n",
    "          \"falseNegativeCount\": \"4172\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99741936,\n",
    "          \"recall\": 0.09053675,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.16604072,\n",
    "          \"truePositiveCount\": \"420\",\n",
    "          \"falseNegativeCount\": \"4219\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9974875,\n",
    "          \"recall\": 0.080620825,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.14921205,\n",
    "          \"truePositiveCount\": \"374\",\n",
    "          \"falseNegativeCount\": \"4265\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9976044,\n",
    "          \"recall\": 0.070273764,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.13131924,\n",
    "          \"truePositiveCount\": \"326\",\n",
    "          \"falseNegativeCount\": \"4313\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99769753,\n",
    "          \"recall\": 0.06014227,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.11346076,\n",
    "          \"truePositiveCount\": \"279\",\n",
    "          \"falseNegativeCount\": \"4360\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99778664,\n",
    "          \"recall\": 0.049795214,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.09486653,\n",
    "          \"truePositiveCount\": \"231\",\n",
    "          \"falseNegativeCount\": \"4408\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9978742,\n",
    "          \"recall\": 0.040525977,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.07789517,\n",
    "          \"truePositiveCount\": \"188\",\n",
    "          \"falseNegativeCount\": \"4451\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.997992,\n",
    "          \"recall\": 0.030394481,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.058995817,\n",
    "          \"truePositiveCount\": \"141\",\n",
    "          \"falseNegativeCount\": \"4498\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9980989,\n",
    "          \"recall\": 0.020262988,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.03972111,\n",
    "          \"truePositiveCount\": \"94\",\n",
    "          \"falseNegativeCount\": \"4545\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.99821925,\n",
    "          \"recall\": 0.010347057,\n",
    "          \"precision\": 1.0,\n",
    "          \"f1Score\": 0.020482184,\n",
    "          \"truePositiveCount\": \"48\",\n",
    "          \"falseNegativeCount\": \"4591\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 0.9987167,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4639\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        },\n",
    "        {\n",
    "          \"confidenceThreshold\": 1.0,\n",
    "          \"precision\": \"NaN\",\n",
    "          \"f1Score\": \"NaN\",\n",
    "          \"falseNegativeCount\": \"4639\",\n",
    "          \"trueNegativeCount\": \"4639\",\n",
    "          \"positionThreshold\": 2147483647\n",
    "        }\n",
    "      ],\n",
    "      \"confusionMatrix\": {\n",
    "        \"row\": [\n",
    "          {\n",
    "            \"exampleCount\": [\n",
    "              3941,\n",
    "              147\n",
    "            ]\n",
    "          },\n",
    "          {\n",
    "            \"exampleCount\": [\n",
    "              293,\n",
    "              258\n",
    "            ]\n",
    "          }\n",
    "        ],\n",
    "        \"displayName\": [\n",
    "          \"1\",\n",
    "          \"2\"\n",
    "        ]\n",
    "      },\n",
    "      \"auRoc\": 0.9743269,\n",
    "      \"logLoss\": 0.20077285\n",
    "    }\n",
    "  }\n",
    "]\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i6T0bzuNJjkY"
   },
   "source": [
    "### [projects.locations.models.modelEvaluations.get](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.models.modelEvaluations/get)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bvhd2ITWU7i7"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vKz1dRGaU7i8"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].get_model_evaluation(\n",
    "    model_evaluation_name=evaluation_slice\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YOHFYa0rU7i8"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ycPBW1ZvU7i8",
    "outputId": "e410231d-89fd-4e58-bde3-3038302fb64c",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(request.__dict__[\"_pb\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264/modelEvaluations/124023939996839080\",\n",
    "  \"annotationSpecId\": \"not available\",\n",
    "  \"createTime\": \"2021-02-28T17:24:31.800758Z\",\n",
    "  \"evaluatedExampleCount\": 4639,\n",
    "  \"classificationEvaluationMetrics\": {\n",
    "    \"auPrc\": 0.89604473,\n",
    "    \"confidenceMetricsEntry\": [\n",
    "      {\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.009267787,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.08771089,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.18867286,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.24711765,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.29667678,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.33910435,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.3722213,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.41141066,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.44597256,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.48088387,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.5,\n",
    "        \"recall\": 0.90515196,\n",
    "        \"precision\": 0.90515196,\n",
    "        \"f1Score\": 0.90515196,\n",
    "        \"falsePositiveRate\": 0.09484803,\n",
    "        \"truePositiveCount\": \"4199\",\n",
    "        \"falsePositiveCount\": \"440\",\n",
    "        \"falseNegativeCount\": \"440\",\n",
    "        \"trueNegativeCount\": \"4199\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.51082355,\n",
    "        \"recall\": 0.9006251,\n",
    "        \"precision\": 0.90688086,\n",
    "        \"f1Score\": 0.90374213,\n",
    "        \"falsePositiveRate\": 0.09247683,\n",
    "        \"truePositiveCount\": \"4178\",\n",
    "        \"falsePositiveCount\": \"429\",\n",
    "        \"falseNegativeCount\": \"461\",\n",
    "        \"trueNegativeCount\": \"4210\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.5395309,\n",
    "        \"recall\": 0.8902781,\n",
    "        \"precision\": 0.9145261,\n",
    "        \"f1Score\": 0.9022392,\n",
    "        \"falsePositiveRate\": 0.083207585,\n",
    "        \"truePositiveCount\": \"4130\",\n",
    "        \"falsePositiveCount\": \"386\",\n",
    "        \"falseNegativeCount\": \"509\",\n",
    "        \"trueNegativeCount\": \"4253\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.569127,\n",
    "        \"recall\": 0.88014656,\n",
    "        \"precision\": 0.92083895,\n",
    "        \"f1Score\": 0.90003306,\n",
    "        \"falsePositiveRate\": 0.07566286,\n",
    "        \"truePositiveCount\": \"4083\",\n",
    "        \"falsePositiveCount\": \"351\",\n",
    "        \"falseNegativeCount\": \"556\",\n",
    "        \"trueNegativeCount\": \"4288\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.59819055,\n",
    "        \"recall\": 0.8700151,\n",
    "        \"precision\": 0.9284564,\n",
    "        \"f1Score\": 0.8982862,\n",
    "        \"falsePositiveRate\": 0.06704031,\n",
    "        \"truePositiveCount\": \"4036\",\n",
    "        \"falsePositiveCount\": \"311\",\n",
    "        \"falseNegativeCount\": \"603\",\n",
    "        \"trueNegativeCount\": \"4328\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.6267493,\n",
    "        \"recall\": 0.86009914,\n",
    "        \"precision\": 0.9350832,\n",
    "        \"f1Score\": 0.8960252,\n",
    "        \"falsePositiveRate\": 0.059711143,\n",
    "        \"truePositiveCount\": \"3990\",\n",
    "        \"falsePositiveCount\": \"277\",\n",
    "        \"falseNegativeCount\": \"649\",\n",
    "        \"trueNegativeCount\": \"4362\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.6483357,\n",
    "        \"recall\": 0.84996766,\n",
    "        \"precision\": 0.9401526,\n",
    "        \"f1Score\": 0.8927884,\n",
    "        \"falsePositiveRate\": 0.05410649,\n",
    "        \"truePositiveCount\": \"3943\",\n",
    "        \"falsePositiveCount\": \"251\",\n",
    "        \"falseNegativeCount\": \"696\",\n",
    "        \"trueNegativeCount\": \"4388\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.68742466,\n",
    "        \"recall\": 0.8404829,\n",
    "        \"precision\": 0.95051193,\n",
    "        \"f1Score\": 0.8921176,\n",
    "        \"falsePositiveRate\": 0.04375943,\n",
    "        \"truePositiveCount\": \"3899\",\n",
    "        \"falsePositiveCount\": \"203\",\n",
    "        \"falseNegativeCount\": \"740\",\n",
    "        \"trueNegativeCount\": \"4436\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.7178758,\n",
    "        \"recall\": 0.83035135,\n",
    "        \"precision\": 0.95630586,\n",
    "        \"f1Score\": 0.8888889,\n",
    "        \"falsePositiveRate\": 0.03793921,\n",
    "        \"truePositiveCount\": \"3852\",\n",
    "        \"falsePositiveCount\": \"176\",\n",
    "        \"falseNegativeCount\": \"787\",\n",
    "        \"trueNegativeCount\": \"4463\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.7454512,\n",
    "        \"recall\": 0.8200043,\n",
    "        \"precision\": 0.9620637,\n",
    "        \"f1Score\": 0.8853718,\n",
    "        \"falsePositiveRate\": 0.032334555,\n",
    "        \"truePositiveCount\": \"3804\",\n",
    "        \"falsePositiveCount\": \"150\",\n",
    "        \"falseNegativeCount\": \"835\",\n",
    "        \"trueNegativeCount\": \"4489\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.7702336,\n",
    "        \"recall\": 0.8105195,\n",
    "        \"precision\": 0.9685729,\n",
    "        \"f1Score\": 0.8825255,\n",
    "        \"falsePositiveRate\": 0.02629877,\n",
    "        \"truePositiveCount\": \"3760\",\n",
    "        \"falsePositiveCount\": \"122\",\n",
    "        \"falseNegativeCount\": \"879\",\n",
    "        \"trueNegativeCount\": \"4517\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.79181457,\n",
    "        \"recall\": 0.80038804,\n",
    "        \"precision\": 0.972244,\n",
    "        \"f1Score\": 0.87798536,\n",
    "        \"falsePositiveRate\": 0.022849752,\n",
    "        \"truePositiveCount\": \"3713\",\n",
    "        \"falsePositiveCount\": \"106\",\n",
    "        \"falseNegativeCount\": \"926\",\n",
    "        \"trueNegativeCount\": \"4533\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.80905926,\n",
    "        \"recall\": 0.7902565,\n",
    "        \"precision\": 0.9755189,\n",
    "        \"f1Score\": 0.873169,\n",
    "        \"falsePositiveRate\": 0.01983186,\n",
    "        \"truePositiveCount\": \"3666\",\n",
    "        \"falsePositiveCount\": \"92\",\n",
    "        \"falseNegativeCount\": \"973\",\n",
    "        \"trueNegativeCount\": \"4547\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.82475233,\n",
    "        \"recall\": 0.7803406,\n",
    "        \"precision\": 0.97574127,\n",
    "        \"f1Score\": 0.86716974,\n",
    "        \"falsePositiveRate\": 0.019400733,\n",
    "        \"truePositiveCount\": \"3620\",\n",
    "        \"falsePositiveCount\": \"90\",\n",
    "        \"falseNegativeCount\": \"1019\",\n",
    "        \"trueNegativeCount\": \"4549\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.84347624,\n",
    "        \"recall\": 0.7702091,\n",
    "        \"precision\": 0.97863597,\n",
    "        \"f1Score\": 0.86200243,\n",
    "        \"falsePositiveRate\": 0.01681397,\n",
    "        \"truePositiveCount\": \"3573\",\n",
    "        \"falsePositiveCount\": \"78\",\n",
    "        \"falseNegativeCount\": \"1066\",\n",
    "        \"trueNegativeCount\": \"4561\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.8612723,\n",
    "        \"recall\": 0.7602932,\n",
    "        \"precision\": 0.9799944,\n",
    "        \"f1Score\": 0.8562758,\n",
    "        \"falsePositiveRate\": 0.015520587,\n",
    "        \"truePositiveCount\": \"3527\",\n",
    "        \"falsePositiveCount\": \"72\",\n",
    "        \"falseNegativeCount\": \"1112\",\n",
    "        \"trueNegativeCount\": \"4567\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.87450475,\n",
    "        \"recall\": 0.75016165,\n",
    "        \"precision\": 0.9811108,\n",
    "        \"f1Score\": 0.8502321,\n",
    "        \"falsePositiveRate\": 0.014442768,\n",
    "        \"truePositiveCount\": \"3480\",\n",
    "        \"falsePositiveCount\": \"67\",\n",
    "        \"falseNegativeCount\": \"1159\",\n",
    "        \"trueNegativeCount\": \"4572\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.8876167,\n",
    "        \"recall\": 0.74003017,\n",
    "        \"precision\": 0.9825415,\n",
    "        \"f1Score\": 0.8442149,\n",
    "        \"falsePositiveRate\": 0.013149385,\n",
    "        \"truePositiveCount\": \"3433\",\n",
    "        \"falsePositiveCount\": \"61\",\n",
    "        \"falseNegativeCount\": \"1206\",\n",
    "        \"trueNegativeCount\": \"4578\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.89729506,\n",
    "        \"recall\": 0.7303298,\n",
    "        \"precision\": 0.98402554,\n",
    "        \"f1Score\": 0.8384063,\n",
    "        \"falsePositiveRate\": 0.011856004,\n",
    "        \"truePositiveCount\": \"3388\",\n",
    "        \"falsePositiveCount\": \"55\",\n",
    "        \"falseNegativeCount\": \"1251\",\n",
    "        \"trueNegativeCount\": \"4584\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9069939,\n",
    "        \"recall\": 0.72041386,\n",
    "        \"precision\": 0.9858407,\n",
    "        \"f1Score\": 0.8324823,\n",
    "        \"falsePositiveRate\": 0.010347057,\n",
    "        \"truePositiveCount\": \"3342\",\n",
    "        \"falsePositiveCount\": \"48\",\n",
    "        \"falseNegativeCount\": \"1297\",\n",
    "        \"trueNegativeCount\": \"4591\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.91544837,\n",
    "        \"recall\": 0.7102824,\n",
    "        \"precision\": 0.987118,\n",
    "        \"f1Score\": 0.8261251,\n",
    "        \"falsePositiveRate\": 0.009269239,\n",
    "        \"truePositiveCount\": \"3295\",\n",
    "        \"falsePositiveCount\": \"43\",\n",
    "        \"falseNegativeCount\": \"1344\",\n",
    "        \"trueNegativeCount\": \"4596\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9246296,\n",
    "        \"recall\": 0.700582,\n",
    "        \"precision\": 0.9896468,\n",
    "        \"f1Score\": 0.8203963,\n",
    "        \"falsePositiveRate\": 0.007329166,\n",
    "        \"truePositiveCount\": \"3250\",\n",
    "        \"falsePositiveCount\": \"34\",\n",
    "        \"falseNegativeCount\": \"1389\",\n",
    "        \"trueNegativeCount\": \"4605\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.932719,\n",
    "        \"recall\": 0.6900194,\n",
    "        \"precision\": 0.99071497,\n",
    "        \"f1Score\": 0.8134689,\n",
    "        \"falsePositiveRate\": 0.006466911,\n",
    "        \"truePositiveCount\": \"3201\",\n",
    "        \"falsePositiveCount\": \"30\",\n",
    "        \"falseNegativeCount\": \"1438\",\n",
    "        \"trueNegativeCount\": \"4609\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.93940556,\n",
    "        \"recall\": 0.6807502,\n",
    "        \"precision\": 0.9921458,\n",
    "        \"f1Score\": 0.80746615,\n",
    "        \"falsePositiveRate\": 0.0053890925,\n",
    "        \"truePositiveCount\": \"3158\",\n",
    "        \"falsePositiveCount\": \"25\",\n",
    "        \"falseNegativeCount\": \"1481\",\n",
    "        \"trueNegativeCount\": \"4614\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9436984,\n",
    "        \"recall\": 0.6704031,\n",
    "        \"precision\": 0.99234205,\n",
    "        \"f1Score\": 0.8002058,\n",
    "        \"falsePositiveRate\": 0.0051735286,\n",
    "        \"truePositiveCount\": \"3110\",\n",
    "        \"falsePositiveCount\": \"24\",\n",
    "        \"falseNegativeCount\": \"1529\",\n",
    "        \"trueNegativeCount\": \"4615\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9486017,\n",
    "        \"recall\": 0.6602716,\n",
    "        \"precision\": 0.992547,\n",
    "        \"f1Score\": 0.7930097,\n",
    "        \"falsePositiveRate\": 0.004957965,\n",
    "        \"truePositiveCount\": \"3063\",\n",
    "        \"falsePositiveCount\": \"23\",\n",
    "        \"falseNegativeCount\": \"1576\",\n",
    "        \"trueNegativeCount\": \"4616\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.95302504,\n",
    "        \"recall\": 0.6501401,\n",
    "        \"precision\": 0.9937397,\n",
    "        \"f1Score\": 0.78603077,\n",
    "        \"falsePositiveRate\": 0.0040957103,\n",
    "        \"truePositiveCount\": \"3016\",\n",
    "        \"falsePositiveCount\": \"19\",\n",
    "        \"falseNegativeCount\": \"1623\",\n",
    "        \"trueNegativeCount\": \"4620\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.95584136,\n",
    "        \"recall\": 0.6400086,\n",
    "        \"precision\": 0.99530673,\n",
    "        \"f1Score\": 0.7790606,\n",
    "        \"falsePositiveRate\": 0.003017892,\n",
    "        \"truePositiveCount\": \"2969\",\n",
    "        \"falsePositiveCount\": \"14\",\n",
    "        \"falseNegativeCount\": \"1670\",\n",
    "        \"trueNegativeCount\": \"4625\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.95881057,\n",
    "        \"recall\": 0.6300927,\n",
    "        \"precision\": 0.9955722,\n",
    "        \"f1Score\": 0.7717492,\n",
    "        \"falsePositiveRate\": 0.002802328,\n",
    "        \"truePositiveCount\": \"2923\",\n",
    "        \"falsePositiveCount\": \"13\",\n",
    "        \"falseNegativeCount\": \"1716\",\n",
    "        \"trueNegativeCount\": \"4626\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.96166563,\n",
    "        \"recall\": 0.62082344,\n",
    "        \"precision\": 0.9961951,\n",
    "        \"f1Score\": 0.76494026,\n",
    "        \"falsePositiveRate\": 0.0023712006,\n",
    "        \"truePositiveCount\": \"2880\",\n",
    "        \"falsePositiveCount\": \"11\",\n",
    "        \"falseNegativeCount\": \"1759\",\n",
    "        \"trueNegativeCount\": \"4628\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.96420497,\n",
    "        \"recall\": 0.61026084,\n",
    "        \"precision\": 0.9961295,\n",
    "        \"f1Score\": 0.75685066,\n",
    "        \"falsePositiveRate\": 0.0023712006,\n",
    "        \"truePositiveCount\": \"2831\",\n",
    "        \"falsePositiveCount\": \"11\",\n",
    "        \"falseNegativeCount\": \"1808\",\n",
    "        \"trueNegativeCount\": \"4628\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.96650803,\n",
    "        \"recall\": 0.5999138,\n",
    "        \"precision\": 0.996063,\n",
    "        \"f1Score\": 0.7488228,\n",
    "        \"falsePositiveRate\": 0.0023712006,\n",
    "        \"truePositiveCount\": \"2783\",\n",
    "        \"falsePositiveCount\": \"11\",\n",
    "        \"falseNegativeCount\": \"1856\",\n",
    "        \"trueNegativeCount\": \"4628\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.96897614,\n",
    "        \"recall\": 0.59042895,\n",
    "        \"precision\": 0.9963623,\n",
    "        \"f1Score\": 0.74147266,\n",
    "        \"falsePositiveRate\": 0.002155637,\n",
    "        \"truePositiveCount\": \"2739\",\n",
    "        \"falsePositiveCount\": \"10\",\n",
    "        \"falseNegativeCount\": \"1900\",\n",
    "        \"trueNegativeCount\": \"4629\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.97092766,\n",
    "        \"recall\": 0.5807286,\n",
    "        \"precision\": 0.99667037,\n",
    "        \"f1Score\": 0.73385996,\n",
    "        \"falsePositiveRate\": 0.0019400733,\n",
    "        \"truePositiveCount\": \"2694\",\n",
    "        \"falsePositiveCount\": \"9\",\n",
    "        \"falseNegativeCount\": \"1945\",\n",
    "        \"trueNegativeCount\": \"4630\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9727514,\n",
    "        \"recall\": 0.5699504,\n",
    "        \"precision\": 0.9969834,\n",
    "        \"f1Score\": 0.7252777,\n",
    "        \"falsePositiveRate\": 0.0017245096,\n",
    "        \"truePositiveCount\": \"2644\",\n",
    "        \"falsePositiveCount\": \"8\",\n",
    "        \"falseNegativeCount\": \"1995\",\n",
    "        \"trueNegativeCount\": \"4631\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9747563,\n",
    "        \"recall\": 0.56046563,\n",
    "        \"precision\": 0.99731493,\n",
    "        \"f1Score\": 0.7176373,\n",
    "        \"falsePositiveRate\": 0.001508946,\n",
    "        \"truePositiveCount\": \"2600\",\n",
    "        \"falsePositiveCount\": \"7\",\n",
    "        \"falseNegativeCount\": \"2039\",\n",
    "        \"trueNegativeCount\": \"4632\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9766838,\n",
    "        \"recall\": 0.549903,\n",
    "        \"precision\": 0.9976535,\n",
    "        \"f1Score\": 0.709005,\n",
    "        \"falsePositiveRate\": 0.0012933821,\n",
    "        \"truePositiveCount\": \"2551\",\n",
    "        \"falsePositiveCount\": \"6\",\n",
    "        \"falseNegativeCount\": \"2088\",\n",
    "        \"trueNegativeCount\": \"4633\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.97776866,\n",
    "        \"recall\": 0.54063374,\n",
    "        \"precision\": 0.99801034,\n",
    "        \"f1Score\": 0.7013423,\n",
    "        \"falsePositiveRate\": 0.0010778185,\n",
    "        \"truePositiveCount\": \"2508\",\n",
    "        \"falsePositiveCount\": \"5\",\n",
    "        \"falseNegativeCount\": \"2131\",\n",
    "        \"trueNegativeCount\": \"4634\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9792647,\n",
    "        \"recall\": 0.53050226,\n",
    "        \"precision\": 0.9979724,\n",
    "        \"f1Score\": 0.6927516,\n",
    "        \"falsePositiveRate\": 0.0010778185,\n",
    "        \"truePositiveCount\": \"2461\",\n",
    "        \"falsePositiveCount\": \"5\",\n",
    "        \"falseNegativeCount\": \"2178\",\n",
    "        \"trueNegativeCount\": \"4634\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9804138,\n",
    "        \"recall\": 0.5203708,\n",
    "        \"precision\": 0.99834573,\n",
    "        \"f1Score\": 0.6841434,\n",
    "        \"falsePositiveRate\": 0.0008622548,\n",
    "        \"truePositiveCount\": \"2414\",\n",
    "        \"falsePositiveCount\": \"4\",\n",
    "        \"falseNegativeCount\": \"2225\",\n",
    "        \"trueNegativeCount\": \"4635\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9814273,\n",
    "        \"recall\": 0.51045483,\n",
    "        \"precision\": 0.9991561,\n",
    "        \"f1Score\": 0.6757027,\n",
    "        \"falsePositiveRate\": 0.0004311274,\n",
    "        \"truePositiveCount\": \"2368\",\n",
    "        \"falsePositiveCount\": \"2\",\n",
    "        \"falseNegativeCount\": \"2271\",\n",
    "        \"trueNegativeCount\": \"4637\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.98243207,\n",
    "        \"recall\": 0.5005389,\n",
    "        \"precision\": 0.9991394,\n",
    "        \"f1Score\": 0.6669539,\n",
    "        \"falsePositiveRate\": 0.0004311274,\n",
    "        \"truePositiveCount\": \"2322\",\n",
    "        \"falsePositiveCount\": \"2\",\n",
    "        \"falseNegativeCount\": \"2317\",\n",
    "        \"trueNegativeCount\": \"4637\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9835001,\n",
    "        \"recall\": 0.49019185,\n",
    "        \"precision\": 0.9995604,\n",
    "        \"f1Score\": 0.6577958,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"2274\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2365\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9844133,\n",
    "        \"recall\": 0.4804915,\n",
    "        \"precision\": 0.9995516,\n",
    "        \"f1Score\": 0.6490028,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"2229\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2410\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.98528725,\n",
    "        \"recall\": 0.47079113,\n",
    "        \"precision\": 0.99954236,\n",
    "        \"f1Score\": 0.6400938,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"2184\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2455\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.98630506,\n",
    "        \"recall\": 0.46001294,\n",
    "        \"precision\": 0.9995316,\n",
    "        \"f1Score\": 0.6300561,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"2134\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2505\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9871925,\n",
    "        \"recall\": 0.44988143,\n",
    "        \"precision\": 0.9995211,\n",
    "        \"f1Score\": 0.6204846,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"2087\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2552\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9877509,\n",
    "        \"recall\": 0.44082776,\n",
    "        \"precision\": 0.99951124,\n",
    "        \"f1Score\": 0.6118175,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"2045\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2594\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9885219,\n",
    "        \"recall\": 0.43048072,\n",
    "        \"precision\": 0.9994995,\n",
    "        \"f1Score\": 0.6017779,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"1997\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2642\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9892192,\n",
    "        \"recall\": 0.4199181,\n",
    "        \"precision\": 0.9994869,\n",
    "        \"f1Score\": 0.5913783,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"1948\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2691\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9897657,\n",
    "        \"recall\": 0.41000214,\n",
    "        \"precision\": 0.9994745,\n",
    "        \"f1Score\": 0.5814735,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"1902\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2737\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9902156,\n",
    "        \"recall\": 0.40051734,\n",
    "        \"precision\": 0.99946207,\n",
    "        \"f1Score\": 0.57186824,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"1858\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2781\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.990671,\n",
    "        \"recall\": 0.3901703,\n",
    "        \"precision\": 0.9994478,\n",
    "        \"f1Score\": 0.5612403,\n",
    "        \"falsePositiveRate\": 0.0002155637,\n",
    "        \"truePositiveCount\": \"1810\",\n",
    "        \"falsePositiveCount\": \"1\",\n",
    "        \"falseNegativeCount\": \"2829\",\n",
    "        \"trueNegativeCount\": \"4638\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99106044,\n",
    "        \"recall\": 0.38025436,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.5509917,\n",
    "        \"truePositiveCount\": \"1764\",\n",
    "        \"falseNegativeCount\": \"2875\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99142385,\n",
    "        \"recall\": 0.36990732,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.5400472,\n",
    "        \"truePositiveCount\": \"1716\",\n",
    "        \"falseNegativeCount\": \"2923\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.991821,\n",
    "        \"recall\": 0.35999137,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.52940243,\n",
    "        \"truePositiveCount\": \"1670\",\n",
    "        \"falseNegativeCount\": \"2969\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99219143,\n",
    "        \"recall\": 0.35007545,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.5186013,\n",
    "        \"truePositiveCount\": \"1624\",\n",
    "        \"falseNegativeCount\": \"3015\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99265563,\n",
    "        \"recall\": 0.3401595,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.50764036,\n",
    "        \"truePositiveCount\": \"1578\",\n",
    "        \"falseNegativeCount\": \"3061\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9930375,\n",
    "        \"recall\": 0.33045915,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.49675956,\n",
    "        \"truePositiveCount\": \"1533\",\n",
    "        \"falseNegativeCount\": \"3106\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9932301,\n",
    "        \"recall\": 0.32054323,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.48547176,\n",
    "        \"truePositiveCount\": \"1487\",\n",
    "        \"falseNegativeCount\": \"3152\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99349546,\n",
    "        \"recall\": 0.31041172,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.47376212,\n",
    "        \"truePositiveCount\": \"1440\",\n",
    "        \"falseNegativeCount\": \"3199\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.993766,\n",
    "        \"recall\": 0.30006468,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.461615,\n",
    "        \"truePositiveCount\": \"1392\",\n",
    "        \"falseNegativeCount\": \"3247\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9940096,\n",
    "        \"recall\": 0.29014874,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.44979113,\n",
    "        \"truePositiveCount\": \"1346\",\n",
    "        \"falseNegativeCount\": \"3293\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9942971,\n",
    "        \"recall\": 0.27980167,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.4372579,\n",
    "        \"truePositiveCount\": \"1298\",\n",
    "        \"falseNegativeCount\": \"3341\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9945464,\n",
    "        \"recall\": 0.26988575,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.42505518,\n",
    "        \"truePositiveCount\": \"1252\",\n",
    "        \"falseNegativeCount\": \"3387\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.994734,\n",
    "        \"recall\": 0.2601854,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.41293192,\n",
    "        \"truePositiveCount\": \"1207\",\n",
    "        \"falseNegativeCount\": \"3432\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9949861,\n",
    "        \"recall\": 0.25026944,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.40034482,\n",
    "        \"truePositiveCount\": \"1161\",\n",
    "        \"falseNegativeCount\": \"3478\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99523896,\n",
    "        \"recall\": 0.2399224,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.38699582,\n",
    "        \"truePositiveCount\": \"1113\",\n",
    "        \"falseNegativeCount\": \"3526\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99545944,\n",
    "        \"recall\": 0.23043759,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.37456203,\n",
    "        \"truePositiveCount\": \"1069\",\n",
    "        \"falseNegativeCount\": \"3570\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9956529,\n",
    "        \"recall\": 0.22009054,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.36077738,\n",
    "        \"truePositiveCount\": \"1021\",\n",
    "        \"falseNegativeCount\": \"3618\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9958653,\n",
    "        \"recall\": 0.21039017,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.34764025,\n",
    "        \"truePositiveCount\": \"976\",\n",
    "        \"falseNegativeCount\": \"3663\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99604857,\n",
    "        \"recall\": 0.20025867,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.33369252,\n",
    "        \"truePositiveCount\": \"929\",\n",
    "        \"falseNegativeCount\": \"3710\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99617386,\n",
    "        \"recall\": 0.19012718,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.31950733,\n",
    "        \"truePositiveCount\": \"882\",\n",
    "        \"falseNegativeCount\": \"3757\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9963188,\n",
    "        \"recall\": 0.18042682,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.3056976,\n",
    "        \"truePositiveCount\": \"837\",\n",
    "        \"falseNegativeCount\": \"3802\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99644965,\n",
    "        \"recall\": 0.17029533,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.29102966,\n",
    "        \"truePositiveCount\": \"790\",\n",
    "        \"falseNegativeCount\": \"3849\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99657893,\n",
    "        \"recall\": 0.16059496,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.27674592,\n",
    "        \"truePositiveCount\": \"745\",\n",
    "        \"falseNegativeCount\": \"3894\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99672806,\n",
    "        \"recall\": 0.1502479,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.2612444,\n",
    "        \"truePositiveCount\": \"697\",\n",
    "        \"falseNegativeCount\": \"3942\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9968369,\n",
    "        \"recall\": 0.14033197,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.24612476,\n",
    "        \"truePositiveCount\": \"651\",\n",
    "        \"falseNegativeCount\": \"3988\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9969655,\n",
    "        \"recall\": 0.13041604,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.23073989,\n",
    "        \"truePositiveCount\": \"605\",\n",
    "        \"falseNegativeCount\": \"4034\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9970963,\n",
    "        \"recall\": 0.12028454,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.21473928,\n",
    "        \"truePositiveCount\": \"558\",\n",
    "        \"falseNegativeCount\": \"4081\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9972023,\n",
    "        \"recall\": 0.110799745,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.19949543,\n",
    "        \"truePositiveCount\": \"514\",\n",
    "        \"falseNegativeCount\": \"4125\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9973033,\n",
    "        \"recall\": 0.100668244,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.18292205,\n",
    "        \"truePositiveCount\": \"467\",\n",
    "        \"falseNegativeCount\": \"4172\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99741936,\n",
    "        \"recall\": 0.09053675,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.16604072,\n",
    "        \"truePositiveCount\": \"420\",\n",
    "        \"falseNegativeCount\": \"4219\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9974875,\n",
    "        \"recall\": 0.080620825,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.14921205,\n",
    "        \"truePositiveCount\": \"374\",\n",
    "        \"falseNegativeCount\": \"4265\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9976044,\n",
    "        \"recall\": 0.070273764,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.13131924,\n",
    "        \"truePositiveCount\": \"326\",\n",
    "        \"falseNegativeCount\": \"4313\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99769753,\n",
    "        \"recall\": 0.06014227,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.11346076,\n",
    "        \"truePositiveCount\": \"279\",\n",
    "        \"falseNegativeCount\": \"4360\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99778664,\n",
    "        \"recall\": 0.049795214,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.09486653,\n",
    "        \"truePositiveCount\": \"231\",\n",
    "        \"falseNegativeCount\": \"4408\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9978742,\n",
    "        \"recall\": 0.040525977,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.07789517,\n",
    "        \"truePositiveCount\": \"188\",\n",
    "        \"falseNegativeCount\": \"4451\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.997992,\n",
    "        \"recall\": 0.030394481,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.058995817,\n",
    "        \"truePositiveCount\": \"141\",\n",
    "        \"falseNegativeCount\": \"4498\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9980989,\n",
    "        \"recall\": 0.020262988,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.03972111,\n",
    "        \"truePositiveCount\": \"94\",\n",
    "        \"falseNegativeCount\": \"4545\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.99821925,\n",
    "        \"recall\": 0.010347057,\n",
    "        \"precision\": 1.0,\n",
    "        \"f1Score\": 0.020482184,\n",
    "        \"truePositiveCount\": \"48\",\n",
    "        \"falseNegativeCount\": \"4591\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 0.9987167,\n",
    "        \"precision\": \"NaN\",\n",
    "        \"f1Score\": \"NaN\",\n",
    "        \"falseNegativeCount\": \"4639\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      },\n",
    "      {\n",
    "        \"confidenceThreshold\": 1.0,\n",
    "        \"precision\": \"NaN\",\n",
    "        \"f1Score\": \"NaN\",\n",
    "        \"falseNegativeCount\": \"4639\",\n",
    "        \"trueNegativeCount\": \"4639\",\n",
    "        \"positionThreshold\": 1\n",
    "      }\n",
    "    ],\n",
    "    \"confusionMatrix\": {\n",
    "      \"row\": [\n",
    "        {\n",
    "          \"exampleCount\": [\n",
    "            3941,\n",
    "            147\n",
    "          ]\n",
    "        },\n",
    "        {\n",
    "          \"exampleCount\": [\n",
    "            293,\n",
    "            258\n",
    "          ]\n",
    "        }\n",
    "      ],\n",
    "      \"displayName\": [\n",
    "        \"1\",\n",
    "        \"2\"\n",
    "      ]\n",
    "    },\n",
    "    \"auRoc\": 0.94022393,\n",
    "    \"logLoss\": 0.20077285\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7y2NlzuBU7jB"
   },
   "source": [
    "## Make batch predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P4U3D4suU7jB"
   },
   "source": [
    "### Make a batch prediction file\r\n",
    "\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oX4J0k7HU7jE",
    "outputId": "118c7235-cbf7-4e31-ed7e-c4f21f868025"
   },
   "outputs": [],
   "source": [
    "! gsutil cat $IMPORT_FILE | head -n 1 > tmp.csv\n",
    "! gsutil cat $IMPORT_FILE | tail -n 10 >> tmp.csv\n",
    "\n",
    "! cut -d, -f1-16 tmp.csv > batch.csv\n",
    "\n",
    "gcs_input_uri = \"gs://\" + BUCKET_NAME + \"/batch.csv\"\n",
    "\n",
    "! gsutil cp batch.csv $gcs_input_uri\n",
    "! gsutil cat $gcs_input_uri\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "Age,Job,MaritalStatus,Education,Default,Balance,Housing,Loan,Contact,Day,Month,Duration,Campaign,PDays,Previous,POutcome\n",
    "53,management,married,tertiary,no,583,no,no,cellular,17,nov,226,1,184,4,success\n",
    "34,admin.,single,secondary,no,557,no,no,cellular,17,nov,224,1,-1,0,unknown\n",
    "23,student,single,tertiary,no,113,no,no,cellular,17,nov,266,1,-1,0,unknown\n",
    "73,retired,married,secondary,no,2850,no,no,cellular,17,nov,300,1,40,8,failure\n",
    "25,technician,single,secondary,no,505,no,yes,cellular,17,nov,386,2,-1,0,unknown\n",
    "51,technician,married,tertiary,no,825,no,no,cellular,17,nov,977,3,-1,0,unknown\n",
    "71,retired,divorced,primary,no,1729,no,no,cellular,17,nov,456,2,-1,0,unknown\n",
    "72,retired,married,secondary,no,5715,no,no,cellular,17,nov,1127,5,184,3,success\n",
    "57,blue-collar,married,secondary,no,668,no,no,telephone,17,nov,508,4,-1,0,unknown\n",
    "37,entrepreneur,married,secondary,no,2971,no,no,cellular,17,nov,361,2,188,11,other\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_models_batchpredict:migration,old"
   },
   "source": [
    "### [projects.locations.models.batchPredict](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.models/batchPredict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DA_NyCpsU7jF"
   },
   "source": [
    "#### Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RDhtOV60U7jG",
    "outputId": "bc185bec-5fde-49b9-dc04-5b764afe1925"
   },
   "outputs": [],
   "source": [
    "input_config = {\n",
    "    \"gcs_source\": {\n",
    "        \"input_uris\": [gcs_input_uri]\n",
    "    }\n",
    "}\n",
    "\n",
    "output_config = {\n",
    "    \"gcs_destination\": {\n",
    "        \"output_uri_prefix\": \"gs://\" + f\"{BUCKET_NAME}/batch_output/\"\n",
    "    }\n",
    "}\n",
    "\n",
    "batch_prediction = automl.BatchPredictRequest(\n",
    "    name=model_id,\n",
    "    input_config=input_config,\n",
    "    output_config=output_config\n",
    ")\n",
    "\n",
    "print(MessageToJson(\n",
    "    batch_prediction.__dict__[\"_pb\"])\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264\",\n",
    "  \"inputConfig\": {\n",
    "    \"gcsSource\": {\n",
    "      \"inputUris\": [\n",
    "        \"gs://migration-ucaip-trainingaip-20210228161105/batch.csv\"\n",
    "      ]\n",
    "    }\n",
    "  },\n",
    "  \"outputConfig\": {\n",
    "    \"gcsDestination\": {\n",
    "      \"outputUriPrefix\": \"gs://migration-ucaip-trainingaip-20210228161105/batch_output/\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DV1xyH62U7jK"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ORWxLnxjU7jL"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].batch_predict(\n",
    "    model_name=model_id, \n",
    "    gcs_input_uris=[gcs_input_uri],\n",
    "    gcs_output_uri_prefix=\"gs://\" + f\"{BUCKET_NAME}/batch_output/\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f-2-0ZboU7jL"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NEa2poQsU7jL",
    "outputId": "c1ef9e7b-d5f1-46b2-aa10-d864c949e7f4"
   },
   "outputs": [],
   "source": [
    "result = request.result()\n",
    "\n",
    "print(MessageToJson(result.__dict__[\"_pb\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fnYRCde3xXie",
    "outputId": "7800b730-ccfb-42f6-b8a9-6a3531e830c7"
   },
   "outputs": [],
   "source": [
    "destination_uri = batch_prediction.output_config.gcs_destination.output_uri_prefix[:-1]\n",
    "\n",
    "! gsutil ls $destination_uri/*\n",
    "! gsutil cat $destination_uri/prediction*/*.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "gs://migration-ucaip-trainingaip-20210228161105/batch_output/prediction-bank_20210228161105-2021-02-28T17:28:50.618755Z/errors_1.csv\n",
    "gs://migration-ucaip-trainingaip-20210228161105/batch_output/prediction-bank_20210228161105-2021-02-28T17:28:50.618755Z/tables_1.csv\n",
    "Age,Job,MaritalStatus,Education,Default,Balance,Housing,Loan,Contact,Day,Month,Duration,Campaign,PDays,Previous,POutcome,errors_Deposit\n",
    "Age,Job,MaritalStatus,Education,Default,Balance,Housing,Loan,Contact,Day,Month,Duration,Campaign,PDays,Previous,POutcome,Deposit_1_score,Deposit_2_score\n",
    "73,retired,married,secondary,no,2850,no,no,cellular,17,nov,300,1,40,8,failure,0.55540502071380615,0.44459491968154907\n",
    "72,retired,married,secondary,no,5715,no,no,cellular,17,nov,1127,5,184,3,success,0.29455980658531189,0.7054402232170105\n",
    "51,technician,married,tertiary,no,825,no,no,cellular,17,nov,977,3,-1,0,unknown,0.478891521692276,0.52110844850540161\n",
    "53,management,married,tertiary,no,583,no,no,cellular,17,nov,226,1,184,4,success,0.36687871813774109,0.63312125205993652\n",
    "23,student,single,tertiary,no,113,no,no,cellular,17,nov,266,1,-1,0,unknown,0.62017822265625,0.37982171773910522\n",
    "37,entrepreneur,married,secondary,no,2971,no,no,cellular,17,nov,361,2,188,11,other,0.78739255666732788,0.21260741353034973\n",
    "25,technician,single,secondary,no,505,no,yes,cellular,17,nov,386,2,-1,0,unknown,0.7512086033821106,0.24879136681556702\n",
    "57,blue-collar,married,secondary,no,668,no,no,telephone,17,nov,508,4,-1,0,unknown,0.86376494169235229,0.13623508810997009\n",
    "71,retired,divorced,primary,no,1729,no,no,cellular,17,nov,456,2,-1,0,unknown,0.47541293501853943,0.52458703517913818\n",
    "34,admin.,single,secondary,no,557,no,no,cellular,17,nov,224,1,-1,0,unknown,0.90285629034042358,0.097143754363059998\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rCxhyYjUU7jb"
   },
   "source": [
    "## Make online predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_models_deploy:migration,old"
   },
   "source": [
    "### [projects.locations.models.deploy](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.models/deploy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oDV0__8yU7jd"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fI2-sB5EU7je"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].deploy_model(\n",
    "    model_name=model_id\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GqwHgW7eU7je"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nZnqUFbZU7jj",
    "outputId": "19dbc64f-1e7c-4dc2-a76b-38c9ad9baefb"
   },
   "outputs": [],
   "source": [
    "result = request.result()\n",
    "\n",
    "print(MessageToJson(result))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "text_models_predict:migration,old"
   },
   "source": [
    "### [projects.locations.models.predict](https://cloud.google.com/automl/docs/reference/rest/v1beta1/projects.locations.models/predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare data item for online prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "INSTANCE = {\n",
    "    \"Age\": '58', \n",
    "    \"Job\": \"managment\", \n",
    "    \"MaritalStatus\": \"married\", \n",
    "    \"Education\": \"teritary\", \n",
    "    \"Default\": \"no\",\n",
    "    \"Balance\": '2143', \n",
    "    \"Housing\": \"yes\", \n",
    "    \"Loan\": \"no\", \n",
    "    \"Contact\": \"unknown\", \n",
    "    \"Day\": '5', \n",
    "    \"Month\": \"may\",\n",
    "    \"Duration\": '261', \n",
    "    \"Campaign\": '1', \n",
    "    \"PDays\": '-1', \n",
    "    \"Previous\": 0, \n",
    "    \"POutcome\": \"unknown\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VSXoe_SjU7jk"
   },
   "source": [
    "#### Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1Lhh4bCKU7jk",
    "outputId": "9cff79af-cc47-4581-fbd7-ffaa5ec89e48"
   },
   "outputs": [],
   "source": [
    "instances_list = [INSTANCE]\n",
    "instances = [ParseDict(s, Value()) for s in instances_list]\n",
    "\n",
    "print(MessageToJson(\n",
    "    automl.PredictRequest(\n",
    "        name=model_id,\n",
    "        payload={\"row\": {\"values\": instances}}            \n",
    "    ).__dict__[\"_pb\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"name\": \"projects/116273516712/locations/us-central1/models/TBL8310014324731019264\",\n",
    "  \"payload\": {\n",
    "    \"row\": {\n",
    "      \"values\": [\n",
    "        {\n",
    "          \"Duration\": \"261\",\n",
    "          \"MaritalStatus\": \"married\",\n",
    "          \"Loan\": \"no\",\n",
    "          \"Education\": \"teritary\",\n",
    "          \"Default\": \"no\",\n",
    "          \"Campaign\": \"1\",\n",
    "          \"Balance\": \"2143\",\n",
    "          \"Contact\": \"unknown\",\n",
    "          \"Previous\": 0.0,\n",
    "          \"Day\": \"5\",\n",
    "          \"Housing\": \"yes\",\n",
    "          \"PDays\": \"-1\",\n",
    "          \"Job\": \"managment\",\n",
    "          \"Month\": \"may\",\n",
    "          \"Age\": \"58\",\n",
    "          \"POutcome\": \"unknown\"\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JBvEfH6BU7jq"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mzOwF0DYU7jr"
   },
   "outputs": [],
   "source": [
    "request = clients[\"tables\"].predict(\n",
    "    model_name=model_id,\n",
    "    inputs=INSTANCE\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4hfPdKUrU7jr"
   },
   "source": [
    "#### Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZL9Nx4gCU7jr",
    "outputId": "7361069f-bd89-48c7-85ee-a93fcc89c35a"
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(request.__dict__[\"_pb\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "datasets_import:migration,new,response"
   },
   "source": [
    "*Example output*:\n",
    "```\n",
    "{\n",
    "  \"payload\": [\n",
    "    {\n",
    "      \"tables\": {\n",
    "        \"score\": 0.9947612,\n",
    "        \"value\": \"1\"\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"tables\": {\n",
    "        \"score\": 0.005238751,\n",
    "        \"value\": \"2\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j14fFDBnxXii"
   },
   "source": [
    "### [projects.locations.models.undeploy](https://cloud.google.com/automl/docs/reference/rest/v1/projects.locations.models/undeploy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wrhN_FT4xXii"
   },
   "source": [
    "#### Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kR-o0CO7xXii"
   },
   "outputs": [],
   "source": [
    "request = clients[\"automl\"].undeploy_model(\n",
    "    name=model_id\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JF1qgjOfxXij"
   },
   "source": [
    "#### Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-VbCkwYyxXij"
   },
   "outputs": [],
   "source": [
    "result = request.result()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SpuhCYjSxXij"
   },
   "source": [
    "#### Response\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ot9hRqTVxXij",
    "outputId": "1d17e1c1-2be9-4f4f-a25b-404ee7465f15"
   },
   "outputs": [],
   "source": [
    "print(MessageToJson(result))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4tj-vFT9xXij"
   },
   "source": [
    "*Example output*:\n",
    "\n",
    "```\n",
    "{}\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bQ-VVaSxJjkd"
   },
   "source": [
    "# Cleaning up\r\n",
    "\r\n",
    "To clean up all GCP resources used in this project, you can [delete the GCP\r\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\r\n",
    "\r\n",
    "Otherwise, you can delete the individual resources you created in this tutorial.\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F5cjBuTKU7jx"
   },
   "outputs": [],
   "source": [
    "delete_dataset = True\n",
    "delete_model = True\n",
    "delete_bucket = True\n",
    "\n",
    "# Delete the dataset using the AutoML fully qualified identifier for the dataset\n",
    "try:\n",
    "    if delete_dataset:\n",
    "        clients['automl'].delete_dataset(name=dataset_id)\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n",
    "# Delete the model using the AutoML fully qualified identifier for the model\n",
    "try:\n",
    "    if delete_model:\n",
    "        clients['automl'].delete_model(name=model_id)\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n",
    "\n",
    "if delete_bucket and 'BUCKET_NAME' in globals():\n",
    "    ! gsutil rm -r gs://$BUCKET_NAME\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "J_croPBoJjkK",
    "MDUAZaN3JjkL",
    "fR9geV9pJjkO",
    "hIHTX-pkJjkO",
    "lbv411XjJjkP",
    "SBxKjsdoU7h0",
    "eNQ_vjVFU7h8",
    "text_datasets_create:migration,old",
    "ozaMOwdNJjkT",
    "vHkuSZxkJjkT",
    "FLqkZMD2JjkT",
    "text_datasets_importdata:migration,old",
    "nacQ0lvTU7iU",
    "NEE3oQPOU7ia",
    "BfEBJzZ_U7ib",
    "zYDxpb87U7ig",
    "DS-tkN-lU7ig",
    "LoBmmTiHU7ii",
    "text_models_create:migration,old",
    "wX-bPrRTU7io",
    "GJu9d2CFU7it",
    "yfufMwAEJjkX",
    "SjYvkggyU7iy",
    "jFRv00YOU7iz",
    "i6T0bzuNJjkY",
    "Bvhd2ITWU7i7",
    "YOHFYa0rU7i8",
    "P4U3D4suU7jB",
    "text_models_batchpredict:migration,old",
    "DA_NyCpsU7jF",
    "DV1xyH62U7jK",
    "f-2-0ZboU7jL",
    "text_models_deploy:migration,old",
    "oDV0__8yU7jd",
    "GqwHgW7eU7je",
    "text_models_predict:migration,old",
    "VSXoe_SjU7jk",
    "JBvEfH6BU7jq",
    "4hfPdKUrU7jr"
   ],
   "name": "Kulwant [UJ.4 OLD] AutoML Tables Regression Classification.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "name": "tf2-2-3-gpu.2-3.m55",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-2-3-gpu.2-3:m55"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
